arcane-os 0.62.0 → 0.64.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/README.md +1 -1
- package/browser-runtime/core/client.mjs +7 -2
- package/docs/reference/README.md +7 -0
- package/docs/reference/cli.md +5 -1
- package/docs/reference/core-client.md +11 -2
- package/docs/reference/core-native-packaging.md +57 -8
- package/docs/reference/core-repositories.md +149 -0
- package/docs/reference/inventory/package-api.json +163 -9
- package/docs/reference/local-ai.md +51 -7
- package/docs/reference/local-image-flux.md +154 -0
- package/docs/reference/local-image-generation.md +12 -0
- package/docs/reference/local-image-sdxl.md +139 -0
- package/docs/reference/local-whisper.md +219 -0
- package/docs/reference/native-speech.md +163 -0
- package/docs/reference/runtime-modules.md +16 -2
- package/docs/reference/sdk-api.md +335 -6
- package/package.json +5 -1
- package/runtime/arcane/components/speech.html +15 -3
- package/runtime/arcane/components/voice-transcription.html +19 -3
- package/runtime/arcane/modules/AI.js +250 -27
- package/runtime/arcane/modules/AIProviderRuntime.js +20 -3
- package/runtime/arcane/modules/AIRuntimeState.js +15 -2
- package/runtime/arcane/modules/ComponentContracts.js +3 -0
- package/schemas/arcane-app.schema.json +53 -2
- package/schemas/arcane-package.schema.json +53 -2
- package/src/core/repositories.mjs +210 -0
- package/src/core/services/speech.mjs +361 -0
- package/src/local-ai/config.mjs +9 -3
- package/src/local-ai/dev.mjs +32 -2
- package/src/local-ai/image-library.mjs +9 -0
- package/src/local-ai/image-models/flux.mjs +31 -0
- package/src/local-ai/image-models/sdxl.mjs +18 -0
- package/src/local-ai/install.mjs +26 -7
- package/src/local-ai/whisper/build.mjs +48 -0
- package/src/local-ai/whisper/config.mjs +61 -0
- package/src/local-ai/whisper/index.mjs +515 -0
- package/src/local-ai/whisper/install.mjs +274 -0
- package/src/local-ai/whisper/native/CMakeLists.txt +98 -0
- package/src/local-ai/whisper/native/NOTICE +18 -0
- package/src/local-ai/whisper/native/README.md +104 -0
- package/src/local-ai/whisper/native/main.cpp +489 -0
- package/src/native/portable-layout.mjs +37 -7
- package/src/native/windows-provider.mjs +39 -15
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
# SDXL Base 1.0 through Core
|
|
2
|
+
|
|
3
|
+
The `sdxl-base-1.0` model definition selects Stability AI's complete SDXL Base
|
|
4
|
+
1.0 checkpoint for text-to-image generation through the shared
|
|
5
|
+
[local image runtime](local-image-generation.md). Applications supply their
|
|
6
|
+
own complete prompts and own the resulting images and saved associations.
|
|
7
|
+
The model uses the shared retained native context, progress, cancellation
|
|
8
|
+
and complete PNG result contract.
|
|
9
|
+
|
|
10
|
+
## Model selection
|
|
11
|
+
|
|
12
|
+
Select this runtime record under `native.localAI` in `arcane-app.json`, or
|
|
13
|
+
the corresponding package projection's `localAI` record:
|
|
14
|
+
|
|
15
|
+
```json
|
|
16
|
+
{
|
|
17
|
+
"runtimes": [
|
|
18
|
+
{
|
|
19
|
+
"id": "stable-diffusion.cpp",
|
|
20
|
+
"version": "master-929-3f8527a",
|
|
21
|
+
"backend": "auto",
|
|
22
|
+
"models": ["sdxl-base-1.0"]
|
|
23
|
+
}
|
|
24
|
+
]
|
|
25
|
+
}
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
The built-in ID supplies the cohesive `SDXL_BASE_1_0_MODEL` definition from
|
|
29
|
+
`src/local-ai/image-models/sdxl.mjs`. The shared service exposes its resource
|
|
30
|
+
metadata through `image.status` and `images.inspect()`.
|
|
31
|
+
|
|
32
|
+
Backend discovery, accelerated-device selection and CPU fallback belong to
|
|
33
|
+
the shared runtime. This model definition leaves `context` empty and adds no
|
|
34
|
+
device or precision override. Available acceleration depends on the selected
|
|
35
|
+
runtime distribution; see the shared guide's platform availability. Model
|
|
36
|
+
selection alone does not establish actual execution on any device.
|
|
37
|
+
|
|
38
|
+
## Prepare the checkpoint and generate an image
|
|
39
|
+
|
|
40
|
+
The single `model` resource is the official
|
|
41
|
+
[sd_xl_base_1.0.safetensors checkpoint](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/462165984030d82259a11f4367a4eed129e94a7b/sd_xl_base_1.0.safetensors)
|
|
42
|
+
at revision `462165984030d82259a11f4367a4eed129e94a7b`. It includes the text
|
|
43
|
+
encoders, denoiser and original VAE. Standalone base generation uses this one
|
|
44
|
+
resource; a refiner, separate VAE, LoRA or Python converter is unnecessary for
|
|
45
|
+
this selection. The upstream model carries the
|
|
46
|
+
[CreativeML Open RAIL++-M license](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/462165984030d82259a11f4367a4eed129e94a7b/LICENSE.md).
|
|
47
|
+
|
|
48
|
+
The existing DBOPFS model store owns the authoritative download. The shared
|
|
49
|
+
[model-assets service](model-assets.md) prepares complete native working files
|
|
50
|
+
when the application loads the model. Installing the native runtime does not
|
|
51
|
+
download a second model copy.
|
|
52
|
+
|
|
53
|
+
This example extends the shared generation guide. It uses an existing `images`
|
|
54
|
+
accessor, DBOPFS-backed `modelStore`, application-selected `workingDirectory`
|
|
55
|
+
and request `signal`:
|
|
56
|
+
|
|
57
|
+
```javascript
|
|
58
|
+
import {createBrowserModelSource} from 'arcane-os/ai/browser-wasm';
|
|
59
|
+
import {prepareCoreModelAssets} from 'arcane-os/ai/core-model-assets';
|
|
60
|
+
|
|
61
|
+
const selected = (await images.inspect({signal})).models.find(
|
|
62
|
+
function selectedSDXL(model) {
|
|
63
|
+
return model.id === 'sdxl-base-1.0';
|
|
64
|
+
}
|
|
65
|
+
);
|
|
66
|
+
const resource = selected.resources.model;
|
|
67
|
+
const source = createBrowserModelSource({
|
|
68
|
+
id: selected.id,
|
|
69
|
+
files: [{name: resource.filename, url: resource.url}]
|
|
70
|
+
});
|
|
71
|
+
const stored = await modelStore.ensure(source, {signal});
|
|
72
|
+
const projection = await prepareCoreModelAssets({
|
|
73
|
+
workingDirectory,
|
|
74
|
+
members: source.files.map(function workingMember(member, index) {
|
|
75
|
+
return {path: member.name, file: stored.files[index]};
|
|
76
|
+
}),
|
|
77
|
+
signal
|
|
78
|
+
});
|
|
79
|
+
|
|
80
|
+
try {
|
|
81
|
+
await images.load({
|
|
82
|
+
model: selected.id,
|
|
83
|
+
assetProjectionId: projection.id,
|
|
84
|
+
resourcePaths: {model: resource.filename},
|
|
85
|
+
signal
|
|
86
|
+
});
|
|
87
|
+
} finally {
|
|
88
|
+
await projection.release();
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
const result = await images.generate({
|
|
92
|
+
model: selected.id,
|
|
93
|
+
prompt: 'An octopus librarian shelving tiny books inside a glass submarine.',
|
|
94
|
+
signal
|
|
95
|
+
});
|
|
96
|
+
// Display result.images using the shared guide's Blob preview example.
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Change the prompt to request another scene. The model defaults to a
|
|
100
|
+
1024-by-1024 canvas; pass `parameters: {width, height}` to select other native
|
|
101
|
+
dimensions. Display or save every returned image using its actual width and
|
|
102
|
+
height. Each result contains the complete PNG Blob.
|
|
103
|
+
|
|
104
|
+
Successful loading retains the working projection in the native model owner.
|
|
105
|
+
Releasing the caller's preparation handle preserves that native retain.
|
|
106
|
+
Repeated generation reuses the loaded context; unload it when the application
|
|
107
|
+
no longer needs it. Follow the shared lifecycle contract for request
|
|
108
|
+
cancellation, state subscriptions, unloading and accessor closure.
|
|
109
|
+
|
|
110
|
+
## Defaults and operation scope
|
|
111
|
+
|
|
112
|
+
Only canvas dimensions are model-specific defaults here. Sampling parameters
|
|
113
|
+
remain owned by the selected native engine and can be supplied through
|
|
114
|
+
`parameters.sample_params`. At `master-929-3f8527a`, SDXL resolves the native
|
|
115
|
+
default sampler to Euler A and scheduler to discrete, with 20 steps and CFG 7.
|
|
116
|
+
These are native defaults; Stability AI's separate base-plus-refiner example
|
|
117
|
+
and Python pipeline have their own sampling configuration.
|
|
118
|
+
|
|
119
|
+
The selected engine has an explicit path for SDXL's embedded original VAE,
|
|
120
|
+
including its native convolution scaling. The model definition leaves that
|
|
121
|
+
treatment with the engine and supplies no guessed weight-type override or
|
|
122
|
+
separate VAE checkpoint. Output quality, speed and device behavior depend on
|
|
123
|
+
the actual selected runtime and generation parameters.
|
|
124
|
+
|
|
125
|
+
The SDK forwards the complete caller prompt to the native API. The engine
|
|
126
|
+
applies its own text conditioning and prompt-weighting syntax; see the shared
|
|
127
|
+
guide for native string-boundary limitations. The SDK adds no avatar wording,
|
|
128
|
+
style instructions or application-specific content.
|
|
129
|
+
|
|
130
|
+
This definition advertises `txt2img`. Whole-image editing, masked editing and
|
|
131
|
+
reference-image editing are separate operations with their own model and
|
|
132
|
+
runtime requirements. This base selection does not advertise those operations.
|
|
133
|
+
|
|
134
|
+
The model resource and standalone use follow the
|
|
135
|
+
[official model card](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/462165984030d82259a11f4367a4eed129e94a7b/README.md).
|
|
136
|
+
Native defaults and embedded-VAE behavior follow the selected engine's
|
|
137
|
+
[request implementation](https://github.com/leejet/stable-diffusion.cpp/blob/master-929-3f8527a/src/pipeline/request.cpp),
|
|
138
|
+
[parameter initialization](https://github.com/leejet/stable-diffusion.cpp/blob/master-929-3f8527a/src/stable-diffusion.cpp)
|
|
139
|
+
and [model construction](https://github.com/leejet/stable-diffusion.cpp/blob/master-929-3f8527a/src/pipeline/model_builders.cpp).
|
|
@@ -0,0 +1,219 @@
|
|
|
1
|
+
# Native Whisper transcription
|
|
2
|
+
|
|
3
|
+
Arcane runs `whisper.cpp` through a persistent native helper. A selected model
|
|
4
|
+
stays loaded across recordings. The helper reports actual model readiness,
|
|
5
|
+
native progress and timestamped segments, and completes cancellation before
|
|
6
|
+
the model or recording files are reused. The browser's existing
|
|
7
|
+
`AI.fetchSTT(audioFile, signal)` and Core `speech.transcribe` routes retain the
|
|
8
|
+
application's selected model and cancellation signal.
|
|
9
|
+
|
|
10
|
+
## Select the runtime and model
|
|
11
|
+
|
|
12
|
+
An application's `arcane-app.json` selects native speech alongside its other
|
|
13
|
+
independent local runtimes:
|
|
14
|
+
|
|
15
|
+
```json
|
|
16
|
+
{
|
|
17
|
+
"native": {
|
|
18
|
+
"localAI": {
|
|
19
|
+
"runtimes": [
|
|
20
|
+
{
|
|
21
|
+
"id": "whisper.cpp",
|
|
22
|
+
"backend": "auto",
|
|
23
|
+
"models": ["whisper-small"]
|
|
24
|
+
}
|
|
25
|
+
]
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The equivalent package configuration is `arcane-package.json`'s `localAI`.
|
|
32
|
+
Existing unrelated descriptor fields remain part of the application's file.
|
|
33
|
+
`models` defaults to an empty array, so selecting the runtime alone acquires no
|
|
34
|
+
model. A single selected model becomes `modelId`; multiple models require an
|
|
35
|
+
explicit selected `modelId` before transcription. The built-in `whisper-small`
|
|
36
|
+
descriptor selects the official `ggerganov/whisper.cpp` `ggml-small.bin` model.
|
|
37
|
+
An explicit model descriptor is `{id, path}` or `{id, url, filename?}`.
|
|
38
|
+
|
|
39
|
+
Default Windows x64 preparation acquires the selected complete upstream
|
|
40
|
+
Whisper 1.9.4/b5130 CUDA 12.4 and CPU distributions, FFmpeg 9.0.2 essentials,
|
|
41
|
+
the selected model, and Arcane's helper asset from the matching SDK numeric
|
|
42
|
+
release. The Windows helper includes its selected Visual C++ release runtime
|
|
43
|
+
DLLs beside the executable. Preparation shares matching installation jobs, observes cancellation,
|
|
44
|
+
and preserves the complete selected runtime trees. Model-selection changes
|
|
45
|
+
can reuse the already prepared runtime and matching models.
|
|
46
|
+
|
|
47
|
+
With `backend: "auto"`, accelerated runtimes are attempted before CPU. Failure
|
|
48
|
+
to load an accelerated runtime falls back to the prepared CPU runtime. If
|
|
49
|
+
native accelerated inference fails, Arcane drains that context and retries the
|
|
50
|
+
complete decoded recording once on CPU with the same model. Cancellation,
|
|
51
|
+
input errors and observer errors do not trigger this retry. Explicit `cuda`,
|
|
52
|
+
`metal` or `cpu` selections retain that chosen backend.
|
|
53
|
+
|
|
54
|
+
During that accepted request's internal recovery, the engine reports
|
|
55
|
+
`state: "recovering"`, `loaded: false`, and `busy: true`, with the same selected
|
|
56
|
+
provider and model. It accepts no new transcription while the old context
|
|
57
|
+
drains and the CPU context loads. Only actual CPU readiness restores `ready`.
|
|
58
|
+
An explicitly unloaded, replaced, or closed model remains a cancellation
|
|
59
|
+
boundary; internal recovery preserves the original request and its signal.
|
|
60
|
+
|
|
61
|
+
`requestedBackend` identifies a preference. `observedBackend` and
|
|
62
|
+
`backendEvidence` report native initialization evidence when available; they
|
|
63
|
+
do not claim that every computation runs on the GPU. No driver or compiler is
|
|
64
|
+
installed on application startup.
|
|
65
|
+
|
|
66
|
+
## Core and development integration
|
|
67
|
+
|
|
68
|
+
Development and generated native Core entries compose a
|
|
69
|
+
[`createSpeechService`](./native-speech.md) with the selected Whisper engine.
|
|
70
|
+
Speech preparation and model loading proceed independently of the page,
|
|
71
|
+
chat and image services. Core publishes retained `speech.state` and per-request
|
|
72
|
+
`speech.progress`; transcription availability follows the selected engine's
|
|
73
|
+
actual loaded state. The shared service remains usable when only STT is
|
|
74
|
+
selected, through `transcriptionAvailable` and `roles.stt`.
|
|
75
|
+
|
|
76
|
+
The native bundle includes the helper, runtime libraries, decoder and selected
|
|
77
|
+
model under its runtime directory beside the application executable. Those
|
|
78
|
+
resources stay in the application bundle. They are not embedded into one
|
|
79
|
+
Windows executable. The compiler and matching upstream headers are build-time
|
|
80
|
+
requirements for the first-party helper only.
|
|
81
|
+
|
|
82
|
+
Applications may continue using the existing Core call:
|
|
83
|
+
|
|
84
|
+
```js
|
|
85
|
+
const transcript = await client.speech.transcribe({
|
|
86
|
+
audioBase64,
|
|
87
|
+
mimeType: 'audio/webm',
|
|
88
|
+
model: 'whisper-small'
|
|
89
|
+
}, {signal});
|
|
90
|
+
console.log(transcript.text);
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
FFmpeg decodes the complete recorded media into mono 16 kHz PCM at the native
|
|
94
|
+
transport boundary. The original recording remains untouched during the
|
|
95
|
+
operation. Arcane imposes no recording-duration cap and does not trim the
|
|
96
|
+
transcript. The upstream inference API's signed-integer sample argument is
|
|
97
|
+
reported honestly if a recording cannot fit that native call; no audio is
|
|
98
|
+
silently shortened. Complete operation files are removed only after decoder
|
|
99
|
+
and native inference cleanup have completed.
|
|
100
|
+
|
|
101
|
+
## Public engine API
|
|
102
|
+
|
|
103
|
+
```js
|
|
104
|
+
import {createWhisperRuntime} from 'arcane-os/local-ai/whisper';
|
|
105
|
+
|
|
106
|
+
const stt = createWhisperRuntime({
|
|
107
|
+
runtime: preparedRuntime,
|
|
108
|
+
modelId: 'whisper-small',
|
|
109
|
+
temporaryDirectory: applicationSpeechDirectory,
|
|
110
|
+
onEvent: reportDiagnostic
|
|
111
|
+
});
|
|
112
|
+
const unsubscribe = stt.subscribe(renderModelState);
|
|
113
|
+
await stt.load({signal});
|
|
114
|
+
try {
|
|
115
|
+
const result = await stt.transcribe({audioBase64, mimeType, model: 'whisper-small'}, {
|
|
116
|
+
signal,
|
|
117
|
+
onProgress: renderTranscriptionProgress
|
|
118
|
+
});
|
|
119
|
+
console.log(result.text, result.segments);
|
|
120
|
+
} finally {
|
|
121
|
+
unsubscribe();
|
|
122
|
+
await stt.close();
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
`createWhisperRuntime(options)` performs no model load. `options.runtime` is a
|
|
127
|
+
prepared descriptor returned by `ensureLocalAIRuntimes`; alternatively,
|
|
128
|
+
`options.prepare({signal, onEvent})` lazily returns that descriptor during
|
|
129
|
+
`load`. `temporaryDirectory` is required and belongs to the application.
|
|
130
|
+
|
|
131
|
+
| Member | Contract |
|
|
132
|
+
| --- | --- |
|
|
133
|
+
| `current()` | Returns `providerId`, selected `modelId`, lifecycle `state`, `loaded`, `busy`, correlated `requestId` or null, requested/observed backend, evidence and any error. |
|
|
134
|
+
| `subscribe(listener, {replay = true, signal} = {})` | Immediately replays current state by default and returns an unsubscribe function. |
|
|
135
|
+
| `load({modelId, signal} = {})` | Prepares the runtime if necessary, then waits for actual native model readiness. With no selected model it remains unloaded. |
|
|
136
|
+
| `transcribe(request, {signal, onProgress, requestId} = {})` | Consumes the complete `audioBase64` recording. Optional `language` defaults to native automatic detection; `translate: true` selects Whisper's translation operation. `model` must match the loaded model. Optional `requestId` carries the caller's existing operation identity through lifecycle snapshots as separate control information; it does not alter the recording or request payload. |
|
|
137
|
+
| `unload()` | Cancels and joins active load/transcription, releases the native context, and permits a later explicit load. |
|
|
138
|
+
| `close()` | Joins shutdown and disposes this engine permanently. Repeated calls share the same shutdown. |
|
|
139
|
+
|
|
140
|
+
One retained model processes one recording at a time; a simultaneous request
|
|
141
|
+
receives `WHISPER_BUSY`. Model changes use `await stt.unload()` followed by
|
|
142
|
+
`stt.load({modelId})`. The `ready` state remains `loaded: true` with `busy: true`
|
|
143
|
+
during ordinary inference. Progress records include the native event plus
|
|
144
|
+
`backend` and `attempt`, so a CPU retry remains distinguishable. Segment text
|
|
145
|
+
is preserved in order. The final result is
|
|
146
|
+
`{text, language, duration, segments: [{index, start, end, text}]}`; times are
|
|
147
|
+
seconds and are native segment timestamps, not invented word timestamps.
|
|
148
|
+
|
|
149
|
+
Full process diagnostics stay with `onEvent` and native process failures.
|
|
150
|
+
Applications should use their selected inspection surface for those details
|
|
151
|
+
and keep ordinary user status concise.
|
|
152
|
+
An error snapshot retains the failed operation's `requestId` after `busy`
|
|
153
|
+
becomes false, allowing the caller to correlate its actual pending response.
|
|
154
|
+
A new request or model load supersedes that identity; unload and close clear
|
|
155
|
+
it. No recording or request payload is retained by this correlation field.
|
|
156
|
+
Progress callbacks may cancel through the supplied controller and return;
|
|
157
|
+
they must not await the same transcription or its shutdown from inside its
|
|
158
|
+
own output callback. Await those operations in the initiating owner instead.
|
|
159
|
+
|
|
160
|
+
## Explicit runtimes and helper builds
|
|
161
|
+
|
|
162
|
+
Windows, Linux and macOS host integration uses the same public engine and
|
|
163
|
+
portable helper source. Automatic archive selection currently provides the
|
|
164
|
+
Windows x64 distribution above. Other targets supply their complete selected
|
|
165
|
+
runtime, decoder and compiled helper paths; their native execution requires
|
|
166
|
+
verification on that actual platform. Android requires a host adapter for
|
|
167
|
+
its process and library lifecycle.
|
|
168
|
+
|
|
169
|
+
An explicit runtime requirement can supply:
|
|
170
|
+
|
|
171
|
+
```js
|
|
172
|
+
{
|
|
173
|
+
id: 'whisper.cpp',
|
|
174
|
+
version: '1.9.4',
|
|
175
|
+
backend: 'auto',
|
|
176
|
+
variants: [
|
|
177
|
+
{backend: 'cuda', root: cudaRuntimeDirectory, libraryDirectory: cudaLibraryDirectory},
|
|
178
|
+
{backend: 'cpu', root: cpuRuntimeDirectory, libraryDirectory: cpuLibraryDirectory}
|
|
179
|
+
],
|
|
180
|
+
helperExecutable,
|
|
181
|
+
helperRoot,
|
|
182
|
+
decoderExecutable,
|
|
183
|
+
decoderRoot,
|
|
184
|
+
models: [{id: 'whisper-small', path: selectedModelPath}],
|
|
185
|
+
modelId: 'whisper-small'
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
`libraryDirectory` belongs to its complete variant root. `decoderExecutable`
|
|
190
|
+
belongs to `decoderRoot` (which defaults to its containing directory).
|
|
191
|
+
`helperExecutable` similarly belongs to `helperRoot`; its complete directory
|
|
192
|
+
tree is retained, including app-local native dependencies.
|
|
193
|
+
Preparation copies those selected resources into its owned installation, and
|
|
194
|
+
native packaging projects their paths relative to the application bundle.
|
|
195
|
+
`helperUrl`, `helperVersion`, and `decoderUrl` allow an explicitly selected
|
|
196
|
+
published distribution instead of local paths. Dependency permissions remain
|
|
197
|
+
the caller's responsibility.
|
|
198
|
+
|
|
199
|
+
```js
|
|
200
|
+
import {buildWhisperHelper} from 'arcane-os/local-ai/whisper/build';
|
|
201
|
+
|
|
202
|
+
const helper = await buildWhisperHelper({
|
|
203
|
+
runtime: {directory: runtimeLibraryDirectory, sourceDirectory: matchingUpstreamSource},
|
|
204
|
+
outputRoot: helperBuildDirectory,
|
|
205
|
+
signal,
|
|
206
|
+
onEvent: reportDiagnostic
|
|
207
|
+
});
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
The builder runs CMake configure/build/install against the selected matching
|
|
211
|
+
public upstream headers and runtime. Optional `cmake`, `generator`,
|
|
212
|
+
`cmakeArgs` and `env` select the existing toolchain. It returns the helper
|
|
213
|
+
`executable`, platform, architecture, runtime directory and complete build
|
|
214
|
+
diagnostics. The helper's documented process contract is in
|
|
215
|
+
[`native/README.md`](../../src/local-ai/whisper/native/README.md).
|
|
216
|
+
On MSVC, CMake's installed runtime-library module adds release CRT files to
|
|
217
|
+
the helper's `bin/` directory. `-DMSVC_REDIST_DIR=...` selects an existing
|
|
218
|
+
redistributable directory when needed. Windows provides the Universal CRT;
|
|
219
|
+
compiler tools and debug runtimes stay outside the application bundle.
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
# Native speech service
|
|
2
|
+
|
|
3
|
+
`createSpeechService({stt, tts, signal} = {})` composes independently selected
|
|
4
|
+
native speech engines into one Core service. It is the named and default export
|
|
5
|
+
from `arcane-os/core/speech`.
|
|
6
|
+
|
|
7
|
+
The host supplies an STT engine, a TTS engine, or both. Construction performs no
|
|
8
|
+
installation, model download, model loading, or inference. Core registration
|
|
9
|
+
exposes `speech.status`, `speech.transcribe`, and `speech.synthesize`; an
|
|
10
|
+
operation whose engine was omitted rejects with `SPEECH_ENGINE_UNAVAILABLE`.
|
|
11
|
+
The optional `signal` controls the service lifetime.
|
|
12
|
+
|
|
13
|
+
```js
|
|
14
|
+
import {createCoreRuntime} from 'arcane-os/core/runtime';
|
|
15
|
+
import {createSpeechService} from 'arcane-os/core/speech';
|
|
16
|
+
|
|
17
|
+
// The host prepares and constructs its selected transcription engine.
|
|
18
|
+
const speech = createSpeechService({stt: transcriptionEngine});
|
|
19
|
+
const core = createCoreRuntime({services: [speech]});
|
|
20
|
+
core.start();
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
This service is the shared integration layer. It does not supply or install a
|
|
24
|
+
Kokoro or Sherpa-ONNX engine. Engine preparation, model resources, device
|
|
25
|
+
selection, supported formats, and native process ownership remain with the
|
|
26
|
+
selected engine and its host composition. Browser-only operation remains
|
|
27
|
+
independent of this native service.
|
|
28
|
+
|
|
29
|
+
## Startup and status
|
|
30
|
+
|
|
31
|
+
`start(context)` subscribes to each configured engine and starts their loads
|
|
32
|
+
independently. It returns immediately, allowing Core and `speech.status` to
|
|
33
|
+
respond while models load. A failure in one role leaves the other role usable.
|
|
34
|
+
Each engine publishes the actual state of its selected model.
|
|
35
|
+
|
|
36
|
+
`current()` and `speech.status` return the same snapshot:
|
|
37
|
+
|
|
38
|
+
| Field | Meaning |
|
|
39
|
+
|---|---|
|
|
40
|
+
| `ready` | Both speech roles are available. A service configured with only STT or only TTS retains `ready: false`. |
|
|
41
|
+
| `transcriptionAvailable` | The STT engine reports a loaded model in `ready` or `running` state, and the service is accepting work. |
|
|
42
|
+
| `synthesisAvailable` | The equivalent independent TTS readiness. |
|
|
43
|
+
| `status` | `created`, `ok`, `closing`, or `closed`. `ok` describes the operational service; model loading and failures are reported per role. |
|
|
44
|
+
| `sttEngine`, `ttsEngine` | The configured engines' `providerId` values, or an empty string for an omitted engine. |
|
|
45
|
+
| `roles.stt`, `roles.tts` | Each engine's complete current state plus the service's `available` boolean. |
|
|
46
|
+
| `closed` | Whether the service has completed its shutdown attempt. Any engine cleanup failure remains observable. |
|
|
47
|
+
|
|
48
|
+
An engine's state includes `providerId`, `modelId`, `state`, `loaded`, and
|
|
49
|
+
`busy`, with its complete error or progress information when present. An
|
|
50
|
+
omitted role has `state: 'unavailable'`, `loaded: false`, and `busy: false`.
|
|
51
|
+
File presence, a spawned process, and a callable bridge do not establish loaded
|
|
52
|
+
model readiness. Existing readiness consumers additionally recognize
|
|
53
|
+
the engine IDs `whisper.cpp` and `kokoro-onnx`; another engine ID requires the
|
|
54
|
+
corresponding consumer integration.
|
|
55
|
+
|
|
56
|
+
## Events and subscriptions
|
|
57
|
+
|
|
58
|
+
`subscribe(listener, {replay = true, signal} = {})` immediately delivers the
|
|
59
|
+
current snapshot by default and returns an unsubscribe function. The optional
|
|
60
|
+
subscription signal removes only that listener. Closed services can still
|
|
61
|
+
replay their final snapshot. Promise rejections from listeners are reported
|
|
62
|
+
through developer diagnostics.
|
|
63
|
+
|
|
64
|
+
Changes are published through the SDK's existing event owner and Core as
|
|
65
|
+
`speech.state`. A Core client observes them with
|
|
66
|
+
`Arcane.events.on('speech.state', listener)`. Core's generic runtime replay
|
|
67
|
+
does not replay custom speech state. A browser consumer should subscribe first,
|
|
68
|
+
then request `Arcane.speech.status()`, retaining any newer event received while
|
|
69
|
+
that initial request is pending.
|
|
70
|
+
|
|
71
|
+
`speech.progress` carries `{requestId, role, status: 'Thinking', progress}`.
|
|
72
|
+
The service emits `progress: {phase: 'accepted'}` synchronously when accepting
|
|
73
|
+
a speech request. Subsequent progress is the selected engine's actual report;
|
|
74
|
+
cancelled requests publish no later progress.
|
|
75
|
+
|
|
76
|
+
## Engine interface
|
|
77
|
+
|
|
78
|
+
Each supplied engine implements:
|
|
79
|
+
|
|
80
|
+
- `current()` returning the actual selected-model state.
|
|
81
|
+
- `subscribe(listener)` with immediate current-state replay and an unsubscribe
|
|
82
|
+
function.
|
|
83
|
+
- `load({signal})`, resolving only after the constructor-selected model has
|
|
84
|
+
loaded. Independent engines may load concurrently.
|
|
85
|
+
- `transcribe(request, {signal, onProgress, requestId})` for STT, or
|
|
86
|
+
`synthesize(request, {signal, onProgress, requestId})` for TTS. `requestId` is
|
|
87
|
+
the actual Core request's control metadata, separate from the unchanged payload.
|
|
88
|
+
- `close()`, cancelling and joining its native work and releasing its resources.
|
|
89
|
+
|
|
90
|
+
The service forwards each complete request and result unchanged. It applies
|
|
91
|
+
no input trimming, content limits, model or voice substitution, transcript
|
|
92
|
+
rewriting, or audio conversion. The engine owns any format decoding,
|
|
93
|
+
phonemization, required model segmentation, output encoding, and supported
|
|
94
|
+
model selection. Necessary processing must preserve complete ordered input
|
|
95
|
+
and output through that engine's public contract.
|
|
96
|
+
|
|
97
|
+
## Cancellation and shutdown
|
|
98
|
+
|
|
99
|
+
The request's `context.signal` reaches its engine together with the service's
|
|
100
|
+
lifetime signal. A request waiting for its role's shared startup load can
|
|
101
|
+
cancel promptly without cancelling another request or the other speech role.
|
|
102
|
+
The retained startup load remains observed by the service. Queued and active
|
|
103
|
+
inference cancellation belongs to the engine; it must stop the actual native
|
|
104
|
+
operation and settle independently of service disposal.
|
|
105
|
+
|
|
106
|
+
This matters because Core cancels and awaits active requests before calling
|
|
107
|
+
service shutdown hooks. An engine that interrupts requests only in `close()`
|
|
108
|
+
would prevent Core from reaching that cleanup hook.
|
|
109
|
+
|
|
110
|
+
`close()` and `drain()` stop acceptance, abort the service lifetime, close both
|
|
111
|
+
engines concurrently, and await their startup and request tasks. Repeated calls
|
|
112
|
+
share one shutdown promise. Actual engine cleanup failures reject with an
|
|
113
|
+
`AggregateError` and remain in role diagnostics. `dispose()` additionally
|
|
114
|
+
releases the service's event subscriptions. New work after shutdown begins
|
|
115
|
+
rejects with `CORE_CLOSING`.
|
|
116
|
+
|
|
117
|
+
## AI built-in native speech readiness
|
|
118
|
+
|
|
119
|
+
The `AI` module's `LOCAL_SPEACH` adapter observes `speech.state` before reading
|
|
120
|
+
`Arcane.speech.status()`. A newer event takes precedence over an outstanding
|
|
121
|
+
initial status response. Each role loads independently: STT does not wait for
|
|
122
|
+
TTS, and unmuting waits for the selected TTS model rather than a callable facade.
|
|
123
|
+
|
|
124
|
+
For this service's role snapshots, readiness requires the exact selected
|
|
125
|
+
`modelId`, its engine identity, `loaded: true`, `available: true`, and a `ready`
|
|
126
|
+
or `running` state. Model loading remains an ordinary cancellable waiting state.
|
|
127
|
+
An engine error, unavailable role, or different selected model rejects the
|
|
128
|
+
adapter load. Later readiness loss or engine/model replacement invalidates the
|
|
129
|
+
AI role and cancels its queued and active requests through the existing provider
|
|
130
|
+
runtime. A new explicit role load or unmute can observe the next ready state.
|
|
131
|
+
|
|
132
|
+
An engine can report `state: 'recovering'`, `loaded: false`, `busy: true`, and
|
|
133
|
+
the continuing Core `requestId` while retrying an already accepted request on
|
|
134
|
+
the same selected provider and model. Availability remains false. The built-in
|
|
135
|
+
STT adapter preserves only its own correlated pending request through this
|
|
136
|
+
state; another caller's recovery supplies no continuation authority. The real
|
|
137
|
+
RPC ID comes from `Arcane.speech.transcribe`'s optional `onRequest` observer.
|
|
138
|
+
The engine retains that ID on a terminal error snapshot until the snapshot is
|
|
139
|
+
superseded, allowing the pending RPC to deliver its actual complete failure
|
|
140
|
+
rather than converting it into cancellation. True unload, close, replacement,
|
|
141
|
+
transport loss and caller cancellation retain their existing ownership.
|
|
142
|
+
|
|
143
|
+
The shared AI observation becomes `recovering` with `loaded: false`, `busy:
|
|
144
|
+
true`, and its existing runtime `operationId`. An error may likewise retain
|
|
145
|
+
`busy: true` until the actual operation settles. New inference still requires
|
|
146
|
+
genuine loaded readiness. The STT provider request context includes a
|
|
147
|
+
`refreshState()` callback which rereads that provider's status only while the
|
|
148
|
+
exact request remains owned; it changes observation, not execution authority.
|
|
149
|
+
Shared speech components retain their own real runtime operation through this
|
|
150
|
+
recovery, keep new dispatch readiness-gated, and preserve native microphone
|
|
151
|
+
capture's separate cancellation and final/interim lifetime.
|
|
152
|
+
|
|
153
|
+
Older fixed-model hosts expose the published aggregate-only `SpeechStatus`
|
|
154
|
+
contract instead of per-model lifecycle. The adapter preserves their independent
|
|
155
|
+
health-based readiness for `whisper-small` with `whisper.cpp` and `kokoro` with
|
|
156
|
+
`kokoro-onnx`, using `status: 'ok'` and the corresponding availability boolean.
|
|
157
|
+
That contract is a one-shot health observation, not evidence of a persistently
|
|
158
|
+
loaded native model or a continuous lifecycle event stream.
|
|
159
|
+
|
|
160
|
+
Cancellation, unload, disposal, and route retirement release adapter-owned
|
|
161
|
+
subscriptions. They do not close the host's shared engines or alter complete
|
|
162
|
+
speech requests and results. Browser and explicitly registered speech providers
|
|
163
|
+
retain their existing lifecycle contracts.
|
|
@@ -1121,6 +1121,14 @@ work. Promise settlement proves only that the provider's exposed request promise
|
|
|
1121
1121
|
completed; provider-specific cancellation acknowledgement remains the selected
|
|
1122
1122
|
provider's boundary.
|
|
1123
1123
|
|
|
1124
|
+
An STT provider's request context also supplies `refreshState()`. An engine
|
|
1125
|
+
lifecycle observer may call it to reread `provider.status()` while that exact
|
|
1126
|
+
request remains active. A recovering or failed pending request is observed as
|
|
1127
|
+
`loaded: false`, `busy: true`, with its existing runtime operation ID; the
|
|
1128
|
+
callback does not make a new request ready or change its supplied payload.
|
|
1129
|
+
After cancellation, unload, disposal, replacement, or settlement it has no
|
|
1130
|
+
effect on the retired request.
|
|
1131
|
+
|
|
1124
1132
|
Direct LLM `request()`, `chat()`, and `stream()` use the same message history,
|
|
1125
1133
|
tool-declaration, emitted-call, all-choice, and ordered parallel-call contracts
|
|
1126
1134
|
as the high-level AI API module, including one nonblank matching result for
|
|
@@ -1229,13 +1237,17 @@ observable without exposing provider transports in application code.
|
|
|
1229
1237
|
|
|
1230
1238
|
Exact exports: `AI_RUNTIME_INTENT_EVENT`, `AI_RUNTIME_PROTOCOL`,
|
|
1231
1239
|
`AI_RUNTIME_ROLES`, `AI_RUNTIME_STARTUP_EVENT`, `AI_RUNTIME_STATES`,
|
|
1232
|
-
`AI_RUNTIME_STATE_EVENT`, `getAIRuntimeState`,
|
|
1240
|
+
`AI_RUNTIME_STATE_EVENT`, `continuesAIRuntimeOperation`, `getAIRuntimeState`,
|
|
1233
1241
|
`publishAIRuntimeRoleState`, `publishAIRuntimeRolesState`,
|
|
1234
1242
|
`requestAIRuntimeIntent`, `startAIRuntime`, `subscribeAIRuntimeIntents`, and
|
|
1235
1243
|
`subscribeAIRuntimeState`.
|
|
1236
1244
|
|
|
1237
1245
|
Each role record is exactly `{role,state,providerId,modelId,localOnly,loaded,
|
|
1238
1246
|
busy,operationId,progress,error}`.
|
|
1247
|
+
`continuesAIRuntimeOperation(previous,current,operationId)` identifies the same
|
|
1248
|
+
already-owned provider/model/runtime operation in a busy, unloaded `recovering`
|
|
1249
|
+
or `error` observation. Pass the actual runtime operation ID, not a component's
|
|
1250
|
+
local counter. It grants neither readiness nor permission to start new work.
|
|
1239
1251
|
`subscribeAIRuntimeState(listener,{signal=null,emitCurrent=true})` installs its
|
|
1240
1252
|
subscription and synchronously replays the current mutable snapshot by default;
|
|
1241
1253
|
`subscribeAIRuntimeIntents(listener,{signal=null})` is future-only. Both return
|
|
@@ -1251,7 +1263,9 @@ transcription-model load.
|
|
|
1251
1263
|
### Availability and normalization
|
|
1252
1264
|
|
|
1253
1265
|
**Cross-host state contract.** States are `unavailable`, `unloaded`, `loading`,
|
|
1254
|
-
`ready`, `unloading`, `error`, and `disposed`.
|
|
1266
|
+
`recovering`, `ready`, `unloading`, `error`, and `disposed`. Recovery remains
|
|
1267
|
+
`loaded: false`; `recovering` and `error` may retain `busy: true` with an actual
|
|
1268
|
+
`operationId` until the pending request settles. Revisions increase monotonically.
|
|
1255
1269
|
The events `arcane-ai-runtime-state`, `arcane-ai-runtime-intent`, and
|
|
1256
1270
|
`arcane-ai-runtime-startup-settled` normalize observation only: receiving one
|
|
1257
1271
|
does not grant a native capability, prove browser support, or load a provider.
|