arcane-os 0.95.0 → 0.97.0

package/docs/reference/model-assets.md

@@ -110,17 +110,18 @@ try {
 }
 ```
 
-`prepare({id,workingDirectory,members,signal,onProgress,refresh=false})` reuses
+`prepare({id,workingDirectory,members,archives,signal,onProgress,refresh=false})` reuses
 a complete native acquisition or fetches the explicitly selected members.
 Every member has its original relative `path` and selected `url`; the caller
 supplies the complete companion-file set. Independent members download
-concurrently. Each complete response stream is written directly to its native
+concurrently, with up to four independent source transfers active. Each complete response stream is written directly to its native
 file, without a whole-model buffer or base64 conversion. No model or runtime
 is selected or downloaded by service startup. The result remains the ready
 projection snapshot with ordered `members: [{path,nativePath}]`.
 
-Complete acquisitions persist under `workingDirectory/model-assets/` across
-release, native unload, service disposal and process restart. Relative working
+Complete acquisitions and interrupted transfer state persist under
+`workingDirectory/model-assets/` across release, native unload, service disposal
+and process restart. Relative working
 directories resolve from `appRoot`. Reuse matches the entire ordered
 `[{path,url}]` selection in that actual store; the operation `id` is a separate
 live lifetime, not the stored selection. Different URLs, relative paths, member
@@ -136,14 +137,193 @@ untouched. Later ordinary preparations reuse the newer completed selection.
 A failed refresh leaves the earlier completed acquisition available. Refresh
 does not change the selected URLs or imply upstream revision discovery.
 
-Progress uses `open`, `download`, `complete`, `reuse` and `ready` phases, with
-`completed`, `total` and `unit:'files'`; member progress additionally includes
-`memberIndex` and `path`. Progress callbacks may return a promise, which the
+Progress includes `open`, `download`, `resume`, `restart`, `complete`, `reuse`
+and `ready`, plus waiting, heartbeat, stalled and terminal interruption states.
+Records carry `state`, a user-facing `message`, `completed`, `total` and
+`unit:'files'`; member progress additionally includes `memberIndex` and `path`.
+Recovery guidance belongs in `recovery`; a complete serialized `error` belongs
+in developer inspection. A stalled state means no recent transfer activity was
+observed, not that the SDK has established an Internet outage. Render active
+transfers without inventing a per-file percentage. Progress callbacks may return a promise, which the
 preparation observes in order. Each caller owns its callbacks; a slow or failed
 callback does not hold another caller's transfer or result. Callbacks must not
 await cleanup of the preparation currently calling them. Callback failures
 reject that caller's preparation. HTTP failures
 retain `url`, `status` and the complete response text in `response`.
+During an active member, repeated activity notifications are emitted at most
+once per second; opening, member completion and terminal phases remain distinct.
+Progress describes files and phases, never transfer volume.
+
+### Selected archive companions
+
+An optional `archives` array acquires exact companion members from an upstream
+uncompressed TAR source through this same acquisition owner:
+
+```javascript
+const projection = await modelAssets.prepare({
+    id: selectedOperationId,
+    workingDirectory: selectedModelDirectory,
+    members: selectedModelFiles,
+    archives: [{
+        url: selectedArchiveURL,
+        format: 'tar',
+        members: [
+            {path: 'tokenizer/model_config.yaml', member: 'model_config.yaml'},
+            {path: 'tokenizer/speakers.json', member: 'speakers.json'}
+        ]
+    }],
+    signal,
+    onProgress
+});
+```
+
+The result orders ordinary members first, followed by each archive's declared
+members. Extraction preserves each complete selected file. TAR names and PAX
+path records select members; a leading `./` is archive path framing. Missing
+selected members reject preparation and identify the missing names. Gzip and
+other archive formats are outside this reader's contract.
+
+When the upstream catalog supplies aligned TAR segments, an archive may include
+`ranges:[{start,end,stopBefore?}]`. The reader requests each inclusive HTTP range
+once and parses its TAR framing; `stopBefore` ends that segment before the
+specified member. Offsets and TAR lengths are solely external transport framing.
+They are never download progress, model readiness criteria or content identity.
+Ranges require the matching HTTP partial-content response; no complete-archive
+fallback is silently substituted. Native reuse identifies archive members by
+their selected URL, archive member name and output path, alongside ordinary
+members, without re-fetching an already completed set.
+
+## Discover and open installed files
+
+`discover({directories,catalog=[],recursive=true,signal})` inspects only the
+explicit directories, without downloading, copying, loading or selecting a
+model. Paths resolve from `appRoot`. It returns:
+
+```javascript
+{
+    directories: [{directory, exists, error}], // error is present only on failure
+    models: [{
+        directory,
+        entryPath,
+        members: [{path, nativePath, available}],
+        missingMembers: [],
+        complete: true,
+        source: 'directory', // or 'catalog' or 'acquisition'
+        catalogId // present for a matching caller-owned catalog entry
+    }]
+}
+```
+
+Ordinary `.gguf` files are candidates. Split names such as
+`moon-00001-of-00003.gguf` form one candidate with all expected members in numeric
+order, including explicit missing members. The first shard is `entryPath`,
+including when a completed acquisition declared only later shards or listed
+them out of order. Acquisition candidates retain all their ordinary and
+companion members alongside these required shards; separate filename groups
+are never inferred to be one model. Symlinked model files and directories are
+followed, with already visited directory targets reused rather than rescanned.
+Discovery does not establish a model's role, runtime compatibility or successful
+inference. Those selections remain with the application and engine owners.
+
+Catalog entries are `{id,members:[{path,url?}],archives?,entryPath?}`. They describe
+an exact companion set and can include nested tokenizer files or multiple model
+files. A discovered catalog candidate reports every expected member, including
+missing companions. Completed native acquisition records are also discoverable;
+when the catalog supplies URLs or archive member names, acquisition matching
+also preserves those recorded source selections. A different pinned source with
+the same filenames is not relabeled as that catalog selection. Path-only catalogs
+and ordinary installed-directory candidates make no source-revision claim.
+Unfinished acquisition attempts are ignored. Discovery never removes either.
+Set `recursive:false` for a directory-only inspection. Missing root directories
+are reported with `exists:false`; other root access failures include their
+complete serialized error. A failure while reading a descendant is surfaced.
+
+`openExisting({id,directory,members:[{path}],signal})` opens a complete selected
+set as an ordinary ready projection. It rechecks every member and reports
+`MODEL_ASSET_MEMBERS_MISSING` with `members` when files are absent. It performs
+no copy or network request. Engines use the same `retain(id)` lifecycle.
+Preparation release, engine unload and service disposal never delete borrowed
+originals. External edits or deletion remain controlled by the files' owner.
+
+The application supplies persistent model directories separately from disposable
+build output. The native launch context exposes its platform-specific
+`modelsDirectory`; apps may additionally inspect selected application directories.
+The model-assets service creates no platform directory policy of its own.
+
+`resolveNativeLaunchContext({appId,context})` defaults `modelsDirectory` to
+one shared location for the operating-system user, independent of the app ID.
+It preserves an explicitly supplied `modelsDirectory` unchanged and creates,
+moves or migrates no files:
+
+| Host | Default shared model directory |
+| --- | --- |
+| Windows | `%LOCALAPPDATA%/Arcane/Models`; the user's `AppData/Local` when that environment variable is absent |
+| macOS | `~/Library/Application Support/Arcane/Models` |
+| Linux | `$XDG_DATA_HOME/Arcane/Models` when absolute, otherwise `~/.local/share/Arcane/Models` |
+| Android or another host adaptation | `<stateRoot>/Models`; the host supplies an explicit shared `modelsDirectory` when its platform allows sharing |
+
+Pass this `modelsDirectory` as the acquisition `workingDirectory` and as one
+discovery directory. Completed files then live beneath its `model-assets`
+subdirectory, separate from replaceable application builds. Additional selected
+application directories can participate in the same discovery call.
+
+```javascript
+import {resolveNativeLaunchContext} from 'arcane-os/core/host';
+
+const {modelsDirectory} = resolveNativeLaunchContext({appId: 'moon-radio'});
+const projection = await modelAssets.prepare({
+    id: selectedOperationId,
+    workingDirectory: modelsDirectory,
+    members: selectedMembers,
+    signal,
+    onProgress: renderPreparationProgress
+});
+```
+
+App settings, profiles, chat history, workspace and IPC retain their app-private
+locations. This default applies to applications using the native launch resolver;
+explicit acquisition directories and other provider-owned model stores keep
+their existing contracts. Discover existing app-local models and open them in
+place rather than copying or removing them.
+
+### Prepare a selected local runtime on demand
+
+`createLocalAIService(configuration,{autoStart:false,...options})` registers the
+existing local-AI owner without preparing or starting its selected runtimes at
+Core startup. The default remains `autoStart:true`, preserving existing startup
+behavior. After the user explicitly requests preparation, invoke
+`client.invoke('localai.prepare', {}, {signal,timeoutMs:0})`. The operation uses
+the configured existing runtime preparation callback and returns the current
+local-AI state; it does not invent a runtime or model selection.
+
+Preparation callback events are published as `localai.preparation` with their
+complete original event payload. Subscribe through the existing Core event
+owner to display actual preparation phases. Concurrent preparation requests
+reuse the owner's in-flight work, and subsequent completed calls reuse its
+prepared state. Runtime and model selection, explicit model acquisition and
+microphone activation remain app-owned; preparation is not an inference or
+microphone operation. Rendering the interface and
+discovering installed files need not wait for runtime preparation.
+
+### Browser discovery and explicit download
+
+`arcane-os/ai/core-model-assets` additionally exports
+`discoverCoreModelAssets({client,directories,catalog,recursive,signal})`,
+`openExistingCoreModelAssets({client,directory,members,signal})`, and
+`downloadCoreModelAssets({client,workingDirectory,members,archives,refresh,signal,onProgress})`.
+`client` defaults to the installed Core client. Discovery returns the native
+record above; opening and downloading return `{id,directory,members,release}`.
+The existing Blob-based `prepareCoreModelAssets` keeps its original contract.
+
+Download progress callbacks receive the complete semantic progress record,
+correlated to this preparation and observed in order. On request failure, queued
+terminal progress is delivered before internal cleanup cancellation and promise
+rejection. Explicit caller cancellation retains its no-late-callback boundary.
+Callback failure cancels the operation. Cancellation joins explicit native cleanup and local callback
+delivery before the helper settles; cleanup failures remain observable.
+Callbacks must not await their own enclosing preparation or cancellation join.
+Applications initiate downloads explicitly and display actual phases, selected
+member paths, completed-file counts and errors in the appropriate surface.
 
 `release(id)` relinquishes preparation ownership without a request signal.
 Same-process callers sharing the actual store, complete source selection and
@@ -151,24 +331,31 @@ refresh mode share an acquisition, including across service instances. They
 keep independent operation IDs, retain handles, cancellation and callbacks.
 Cancelling, releasing or disposing one interested caller leaves the others
 running. The final interested caller aborts the acquisition and joins its
-streams, writes and closes before removing only its unfinished attempt. Original
+streams, writes and closes while preserving completed members and resumable
+partial transfers. Original
 failures and cleanup failures remain observable. A completed acquisition stays
 available even if a caller cancels or its callback fails after completion.
 
-The service records completion only after every selected response and file
-close finishes. That record is ordinary cache state describing the selected
-sources and completed acquisition. A missing original member requires another
-complete acquisition. Failed attempts are removed after their owned I/O drains;
-there is no partial-transfer resumption. An abruptly interrupted process can
-leave an unfinished directory, which later calls ignore without sweeping it.
-There is no automatic migration, eviction or removal of completed acquisitions.
-
-Independent processes use disjoint attempt directories and may fetch the same
-selection concurrently. They do not overwrite each other's files or coordinate
-a cross-process acquisition lock. Concurrent completions can produce multiple
-reusable sets; either complete result may be selected when their completion
-order ties. Applications may retain their existing shared Core owner without
-introducing an additional host or application-local cache copy.
+The service promotes each file from its partial-transfer location only after
+the complete response and file close finish. It records whole-acquisition
+completion after every selected member is complete. These records describe
+ordinary transfer and cache state, not content hashes or admission receipts.
+Retry reuses completed members and resumes saved ordinary-file transfers with
+HTTP Range and available HTTP continuation metadata. HTTP offsets and validators
+remain transport-local and never become progress or model policy. If the source
+cannot continue the saved representation, the SDK reports a restart of that
+member; completed siblings remain available. Selected TAR companions preserve
+completed files; an interrupted member may require replaying its TAR segment to
+restore the archive parser's framing. No full-archive substitute is selected.
+
+Processes selecting the same ordered sources and actual store coordinate a
+short selection operation and one transfer owner. Another process waits with
+observable status rather than starting a duplicate transfer. Cancelling a
+waiting request stops its wait; it does not cancel the other process. If a
+transfer owner exits, a later contender can reclaim its recorded operation and
+continue retained data. This requires no additional application server, copied
+cache implementation or new shared Core process. Existing completed models and
+unrelated attempts are never swept or migrated automatically.
 
 ```javascript
 import {createModelAssetService} from 'arcane-os/core/model-assets';
@@ -186,7 +373,8 @@ try {
 ```
 
 `createModelAssetService({appRoot})` returns a Core service with `current()`,
-`prepare(options)`, `retain(id)`, `release(id)` and `dispose()`. `retain()` accepts only a completed projection
+`discover(options)`, `openExisting(options)`, `prepare(options)`, `retain(id)`,
+`release(id)` and `dispose()`. `retain()` accepts only a completed projection
 and returns `{id,directory,members,release}`. A native engine takes this handle
 before loading and releases it after actual unload, failed-load cleanup or
 worker exit. A rejected inference/load promise alone does not establish native
@@ -249,6 +437,10 @@ for source selection, cancellation and complete row/output contracts.
 | Method | Parameters | Result |
 | --- | --- | --- |
 | `modelAssets.open` | `{id,workingDirectory,members:[{path}]}` | Preparing projection and native member paths |
+| `modelAssets.discover` | `{directories,catalog?,recursive?}` | Directory status and complete/incomplete model candidates |
+| `modelAssets.openExisting` | `{id,directory,members:[{path}]}` | Ready retained projection of existing originals |
+| `modelAssets.prepare` | `{id,workingDirectory,members,archives?,refresh?}` | Ready persistent acquisition; request-scoped `modelAssets.progress` emits `{id,progress}` |
+| `modelAssets.cancelPrepare` | `{id}` | Cancel and join the matching native preparation, then release its preparation ownership |
 | `modelAssets.write` | `{id,memberIndex,contentBase64}` | `{id,memberIndex,written:true}` after the native write |
 | `modelAssets.complete` | `{id}` | Ready projection after all files close |
 | `modelAssets.release` | `{id}` | Released preparation ownership; native uses may retain the files |
@@ -258,6 +450,10 @@ The browser helper owns this transport sequence. Correlation IDs identify
 operations; they are unrelated to content identity. Original model files remain
 separate and complete. A retry creates a new working projection from the stored
 assets; it does not redownload them or replace the authoritative DBOPFS record.
+`cancelPrepare` and `release` have service lifetime so cancellation of the
+original request cannot cancel its cleanup control. Native preparation follows
+the original request signal, including connection cancellation; raw RPC callers
+join `cancelPrepare` when cancelling and release any successful unused projection.
 
 The implementation uses browser streams, the existing portable Core JSON
 transport and Node filesystem operations. Native paths and cleanup follow the

package/docs/reference/native-speech.md

@@ -1,6 +1,6 @@
 # Native speech service
 
-`createSpeechService({stt, tts, signal} = {})` composes independently selected
+`createSpeechService({stt, tts, signal, autoLoad = true} = {})` composes independently selected
 native speech engines into one Core service. It is the named and default export
 from `arcane-os/core/speech`.
 
@@ -50,6 +50,13 @@ independently. It returns immediately, allowing Core and `speech.status` to
 respond while models load. A failure in one role leaves the other role usable.
 Each engine publishes the actual state of its selected model.
 
+Use `autoLoad: false` when the host first needs the user to select or download
+model files. Startup still subscribes and exposes the ordinary unloaded state;
+it does not call either engine's `load()`. Explicit `speech.load({role,
+...selection})` activates the prepared role afterward. The default remains
+`true` for existing hosts. This option changes service startup, not the selected
+engine's missing-model errors or inference readiness requirement.
+
 `current()` and `speech.status` return the same snapshot:
 
 | Field | Meaning |
@@ -66,9 +73,10 @@ An engine's state includes `providerId`, `modelId`, `state`, `loaded`, and
 `busy`, with its complete error or progress information when present. An
 omitted role has `state: 'unavailable'`, `loaded: false`, and `busy: false`.
 File presence, a spawned process, and a callable bridge do not establish loaded
-model readiness. Existing readiness consumers additionally recognize
-the engine IDs `whisper.cpp` and `kokoro-onnx`; another engine ID requires the
-corresponding consumer integration.
+model readiness. The shared AI adapter follows the selected engine identity
+and model in these per-role snapshots, including Nemotron and Magpie. Only the
+older aggregate-only fallback retains fixed `whisper.cpp`/`whisper-small` and
+`kokoro-onnx`/`kokoro` expectations.
 
 Each role also reports `supportsStreaming`: STT requires the configured
 engine's `openStream()` method, and TTS requires its `synthesizeStream()` method.
@@ -288,6 +296,122 @@ share one shutdown promise. Actual engine cleanup failures reject with an
 releases the service's event subscriptions. New work after shutdown begins
 rejects with `CORE_CLOSING`.
 
+## Selected NeMo model acquisition
+
+`NEMO_SPEECH_MODELS`, `prepareNemotronModels`, and `prepareMagpieModels` are
+named exports from `arcane-os/local-ai/nemotron`. Importing them performs no
+network access, installation, model loading, or inference. The catalog records
+NVIDIA's selected [NeMo-Speech.cpp v0.2.0 model index](https://github.com/NVIDIA/NeMo-Speech.cpp/blob/v0.2.0/models/index.json):
+
+| Role | Complete default selection |
+|---|---|
+| `stt` | Nemotron 3.5 streaming ASR Q8_0 and Nemotron 3 Diarization Q8_0. |
+| `tts` | Magpie multilingual v2607 F16, NanoCodec 22 kHz F16 decoder, and all ten matching tokenizer/configuration/speaker-map members from the same Magpie revision. |
+
+Each role exposes `{modelId, providerId, name, members, resourcePaths}`.
+`members` contains ordered `{path, url}` file sources. TTS also exposes
+`archives`, with the upstream uncompressed `.nemo` TAR source, its selected
+transport ranges, and exact `{path, member}` tokenizer entries. The shared
+[model-assets service](model-assets.md) owns discovery, acquisition, progress,
+cancellation, persistence, and projection lifetime; these helpers do not create
+another downloader or store. Tokenizer files retain their complete original
+contents. The selected tokenizer configuration and its referenced JSON speaker
+map continue to own voice IDs.
+
+```js
+import {createNemotronRuntime, prepareNemotronModels} from 'arcane-os/local-ai/nemotron';
+import {createSpeechService} from 'arcane-os/core/speech';
+
+const stt = createNemotronRuntime({runtime, modelAssets, gpu: -1});
+const speech = createSpeechService({stt, autoLoad: false});
+// Run this only for the user's explicit model preparation action.
+const prepared = await prepareNemotronModels({
+  modelAssets,
+  workingDirectory: modelsDirectory,
+  directories: [modelsDirectory, appModelsDirectory],
+  signal,
+  onProgress: showModelProgress
+});
+try {
+  await stt.load({
+    assetProjectionId: prepared.assetProjectionId,
+    resourcePaths: prepared.resourcePaths,
+    signal
+  });
+} finally {
+  await prepared.release();
+}
+```
+
+Both preparation helpers accept `{modelAssets, workingDirectory, directories,
+id, signal, onProgress, refresh = false}`. With `directories`, they ask the
+shared owner for the first complete matching catalog selection in those roots
+before downloading. `openExisting()` borrows its original files and never
+deletes them. Incomplete selections stay visible through the owner's discovery
+results and are not mistaken for a complete model. When no complete match is
+available, the existing acquisition path prepares the complete selected set;
+its completed members and closed partial files remain available after an
+interruption or cancellation. Explicit retry reuses those members and resumes
+ordinary HTTP transfers when the source accepts their conditional range;
+otherwise only that member restarts with an explicit progress event. TAR
+segments replay while completed tokenizer companions remain reusable.
+Matching cross-process selections wait for the actual transfer owner, then
+reuse or resume its recorded attempt; unrelated downloads remain concurrent.
+See [native acquisition and recovery](model-assets.md#native-ownership) for the
+shared owner's complete contract. Explicit `refresh: true` requests a
+fresh acquisition. The host chooses the roots: on Windows, Linux and macOS its
+portable native launch-context resolver defaults `modelsDirectory` to the
+operating-system user's shared `Arcane/Models` directory, independently of
+app-private state and workspace. Explicit directory overrides remain unchanged;
+other adapted hosts supply their shared directory explicitly or retain the
+resolver's app-state fallback. This location applies to services using that
+resolver value, not every provider store. Resolution moves no existing files.
+An application may include its own `models` directory and prior app-local
+model locations among the discovery roots, borrowing those originals in place.
+
+The result is `{assetProjectionId, directory, members, resourcePaths, release}`.
+`release()` is idempotent and releases preparation ownership. Each engine
+retains the projection synchronously when `load()` accepts it, so the caller
+may release its preparation after starting that load. Nemotron accepts
+`modelAssets`, `assetProjectionId`, and `resourcePaths` in its constructor and
+`assetProjectionId`/`resourcePaths` in `load()`, with roles `{model, diarization}`.
+Magpie uses `{model, codec, tokenizerConfig}`. Native file paths still work
+through the engines' existing options. Nemotron requires unloading before
+changing an active selection; its retained files are released only after
+native work and helper exit have joined. Load failures also join asset cleanup,
+and cleanup failures remain observable.
+
+`onProgress` receives ordered, awaited shared-owner phase and file-completion
+events. Native acquisition includes `state`, plain `message`, file counts and
+optional member, reason, recovery, error or other-owner activity. Its five-second
+heartbeat and fifteen-second no-source-activity `stalled` state describe actual
+transfer observations, not an inferred outage or percentage. `interrupted`,
+`cancelled`, `failed` and `ready` are distinct terminal states. Direct
+installed-directory reuse retains the existing `reuse`/`ready` records with
+`completed`, `total` and `unit:'files'`.
+
+Signal cancellation stops that caller's preparation interest; another interested
+caller may continue the shared acquisition. The final cancellation joins I/O
+and retains resumable data; cancelled subscribers keep their existing callback
+boundary. An abort after projection creation releases preparation ownership
+before rejecting. Engine load cancellation remains separate: cancelling the
+caller's wait does not cancel a shared activation; explicit engine unload/close
+owns that cancellation.
+
+The default GGUF selections are individual complete upstream files. The shared
+owner also discovers ordered GGUF shard sets and reports their missing members;
+HTTP range transfers and tokenizer TAR ranges are transport mechanisms, not
+additional model shards. A discovered shard set alone does not establish that a
+selected native engine can load that model. The engine remains the authority
+for model compatibility and actual readiness.
+
+The [upstream native runtime](https://github.com/NVIDIA/NeMo-Speech.cpp/blob/v0.2.0/docs/install.md)
+provides Windows, Linux, and macOS distributions. Model files and that runtime
+remain separate upstream assets, not vendored SDK payloads. This acquisition
+integration received source review only; no download, inference, or new platform
+execution is claimed here. See the [Nemotron](native-nemotron.md) and
+[Magpie](native-magpie.md) contracts for native preparation and execution evidence.
+
 ## Native Kokoro engine
 
 `createNativeKokoroRuntime` is the named and default export from