arcane-os 0.57.0 → 0.59.0

package/docs/reference/diarization.md

@@ -0,0 +1,206 @@
+# Nemotron 3 speaker diarization
+
+The native diarization API uses the NVIDIA `Nemotron-3-Diarization.q8_0.gguf`
+model through the official NeMo-Speech.cpp 0.2.0 CPU libraries. It returns
+generic, one-based speaker labels and timed segments, independently of speech
+recognition. A speaker label does not identify a person. The model supports
+eight speaker channels; the helper reports the loaded model's actual channel
+count and output cadence.
+
+## Runtime and helper
+
+Select the `nemo-speech` runtime through the existing local-AI installer. Keep
+its complete upstream `bin`/`lib`/`include` and CMake layout. Model storage is
+separate and caller-selected. Obtain the selected GGUF from the
+[official NVIDIA model repository](https://huggingface.co/nvidia/Nemotron-3-Diarization).
+This API does not select an ASR model, install Python, or enable a GPU backend.
+
+`buildDiarizationHelper({runtime,outputRoot,signal,onEvent})` builds the
+first-party helper for the current host with installed CMake and its C++17
+toolchain. `runtime.cmakeDirectory` is the installer-discovered CMake package
+directory, passed as `NeMoSpeech_DIR` without assuming an archive root. The result
+is `{platform,architecture,root,executable,diagnostics}`; diagnostics retain the
+complete configure/compile/install process results. The installed helper is under
+`outputRoot/bin`; its build directory is under `outputRoot/build`. The helper
+links the upstream `NeMoSpeech::Diarization` component and is separate from
+the downloaded runtime. No compiler is downloaded by this function.
+
+Official 0.2.0 CPU archives cover Windows x64, Linux x64/ARM64, and macOS
+x64/ARM64. Linux requires glibc 2.31 or later. Android needs a host-adapted
+runtime/helper; this release's official inventory does not supply an Android
+or Windows ARM64 archive. Source portability is distinct from execution on an
+actual platform.
+
+The public Node flow uses the existing installer and one first-party helper:
+
+```javascript
+import {ensureLocalAIRuntimes} from 'arcane-os/local-ai';
+import {buildDiarizationHelper} from 'arcane-os/diarization/build';
+import {createDiarization} from 'arcane-os/diarization';
+
+const [runtime] = await ensureLocalAIRuntimes({
+    runtimes: [{id: 'nemo-speech', version: '0.2.0'}],
+    directory: selectedRuntimeDirectory
+});
+const helper = await buildDiarizationHelper({runtime, outputRoot: selectedHelperDirectory});
+```
+
+Keep the selected directories and model path under their application/host
+owner. A native artifact can place the first-party executable under
+`runtime/diarization/bin`, independently of the complete upstream library tree
+under `runtime/local-ai/nemo-speech`.
+
+## Retained model and audio streams
+
+`createDiarization({executable,modelPath,runtime,onEvent})` returns immediately
+and starts its owned helper. Supply absolute helper/model paths and the
+installer's absolute `runtime.binaryDirectory`/`runtime.libraryDirectory` paths. They configure the
+child's platform library search path; the parent environment is unchanged.
+
+The returned model has `ready`, `completion`, `current()`,
+`subscribe(listener,{emitCurrent=true,signal})`, `openStream(options)`,
+`diarize(options)`, and `close()`. `ready` resolves when the actual model is
+loaded; a failed load rejects. `completion` covers the complete helper lifetime.
+The sticky state is `loading`, `ready`, `closing`, `closed`, or `error`.
+Subscribe to this owner instead of polling or delaying unrelated UI startup.
+
+```javascript
+const diarization = createDiarization({
+    executable: helper.executable,
+    modelPath: selectedModelPath,
+    runtime
+});
+
+const stream = await diarization.openStream({
+    sampleRate: 16000,
+    onUpdate: function showSpeakers(result) {
+        renderSpeakerTimeline(result.segments);
+    }
+});
+
+await stream.push(firstMonoSamples);
+await stream.push(nextMonoSamples);
+const final = await stream.finish();
+await diarization.close();
+```
+
+`push(audio)` accepts mono `Float32Array` samples and preserves their float32
+representation. The caller owns decoding an encoded recording and choosing
+its mono channel; this API does not alter the source recording, mix channels,
+normalize amplitudes, or capture a microphone. Keep the supplied array stable
+until `push` settles. Each stream accepts one fixed sample rate: 8000–96000 Hz,
+or `0` for model-rate samples. The default is 16000 Hz. Upstream owns necessary
+resampling. Await each push for backpressure; independent streams share the
+retained model without sharing speaker/session state. Upstream serializes
+model compute internally.
+
+`openStream({sampleRate,onUpdate,onProbabilities,signal})` returns a stream with
+`push`, `finish`, `cancel`, `completion`, and `current()`. A push resolves to
+its latest segment snapshot, or `null` until native frames are available. `finish()`
+flushes all remaining audio and resampler output, returns the complete final
+result, and releases the native stream. It is idempotent. `current()` retains
+only the latest successful snapshot. Results have:
+
+```text
+{final, speakers, secondsPerFrame, frameCount,
+ segments:[{speaker,startTime,endTime}, ...]}
+```
+
+Times are seconds from the start of that caller's stream. Intermediate segment
+snapshots may revise earlier segment boundaries. Replace the displayed
+snapshot; do not append whole snapshots as new segments. Final results include
+all upstream segments, including those finalized before upstream probability
+compaction. The API adds no diarization thresholds or segment filtering.
+
+For recorded mono samples, `await diarization.diarize({audio,sampleRate,signal,
+onUpdate,onProbabilities})` uses the same complete streaming path, then finishes
+and closes that stream. It does not use upstream's length-constrained stateless
+offline operation.
+
+## Probabilities, callbacks, and cancellation
+
+Supply `onProbabilities` to receive every available probability frame as
+`{startFrame,values,speakers,secondsPerFrame}`. Values are frame-major, with one
+value per speaker channel per frame. Frame indices are absolute within the
+stream. The adapter reads between 160 ms input pushes, before upstream's old
+probability buffer can be compacted. It reports an error if an upstream gap is
+ever observed; it does not manufacture missing values. Without this callback,
+the helper does not request raw probabilities.
+
+Callbacks are awaited and their failures propagate. The caller owns retaining
+probability history, audio, and any earlier snapshots it needs. The SDK does
+not accumulate a second transcript of continuous protocol output. `onEvent`
+receives ordinary process lifecycle events and complete stderr diagnostics;
+raw stdout protocol belongs to the result callbacks instead. Keep engineering
+diagnostics in a selected developer surface, outside ordinary UI/chat history.
+
+`cancel(reason)` and the stream's abort signal stop subsequent pushes and
+suppress later result commits for that stream. Cancellation closes the stream
+after native commands already ahead of its close have returned; the C ABI
+does not offer an immediate interrupt. Other streams keep their own state.
+An already-running caller callback finishes under its caller's ownership.
+`cancel()` can be awaited inside a callback. Await the same stream's `push()`,
+`finish()`, or model `close()`
+only after the callback returns; a recursive wait reports
+`DIARIZATION_CALLBACK_WAIT` instead of deadlocking. Stream `completion`
+retains finish/cancellation cleanup failures.
+
+`close()` stops new streams, drains in-flight openings and accepted stream work,
+finishes the remaining streams, closes helper stdin, and waits for native
+cleanup/process exit. The model always outlives its streams. Complete failures
+from an operation and its cleanup remain available together.
+
+## Core service
+
+Import the named or default factory from `arcane-os/core/diarization`.
+
+`createDiarizationService({executable,modelPath,runtime,onEvent},launchContext)`
+is a synchronous `native.services` factory. Relative executable/model paths
+resolve against explicit `launchContext.workspaceRoot`, otherwise the native
+launcher's `appRoot` (or the current directory for standalone factories). Model loading is lazy at the
+first diarization operation, not a prerequisite for Core startup. The returned
+service implements `drain()` and `dispose()` and the following methods:
+
+| Method | Input / result |
+| --- | --- |
+| `diarization.status` | Current model lifecycle snapshot. |
+| `diarization.load` | Load the configured model; return readiness. |
+| `diarization.open` | `{sampleRate,probabilities}` → `{streamId,sampleRate}`. |
+| `diarization.push` | `{streamId,audio}` → current timed segment snapshot. |
+| `diarization.finish` | `{streamId}` → complete final result, releasing the stream. |
+| `diarization.cancel` | `{streamId}` → cleanup completion for that stream. |
+| `diarization.recording` | `{audio,sampleRate}` → complete final result. |
+
+Core audio is `{encoding:'f32le',data:BASE64}`: transport encoding of the exact
+little-endian float32 samples, with control metadata outside the audio. The
+service emits `diarization.state`, `diarization.update` with `{streamId,result}`,
+and optional `diarization.probabilities` with the stream ID and frame data.
+Subscribe before opening a stream. Session operations have service lifetime;
+renderer request cancellation does not discard accepted audio. Call the
+explicit cancel method for a session. A recording operation has request
+lifetime and its cancellation closes only that recording's stream.
+
+## Shared process transport
+
+The existing internal `runProcess` owner accepts scalar input as before and
+also an `AsyncIterable<string|Uint8Array>`. Iteration is backpressured by the
+child's stdin writes; iterable completion sends EOF, including in `close-input`
+mode. Cancellation/early child exit calls the iterator's `return()` and waits
+for input cleanup. The producer must make a pending `next()` settle when it is
+returned/cancelled. The diarization-owned producer does this explicitly.
+
+`onOutput({stream,chunk})` receives complete decoded UTF-8 chunks with original
+delimiters and is awaited per stream. It is not a binary stdout interface.
+`captureOutput` and `emitOutputEvents` default to `true`; each also accepts
+`{stdout,stderr}` overrides. Disabling capture requires an output consumer and
+returns `null` for that process-result field; it does not return a shortened
+transcript. Disabling output events leaves those events absent. Existing
+scalar input, full text results and ordinary event defaults remain unchanged.
+Failed output callbacks retain their complete input chunk and cause in the
+process error, including when successful output is caller-owned.
+
+## Upstream contracts
+
+The implementation follows the [stable C ABI](https://github.com/NVIDIA/NeMo-Speech.cpp/blob/v0.2.0/include/nemo_speech/diar.h),
+[SDK linkage guide](https://github.com/NVIDIA/NeMo-Speech.cpp/blob/v0.2.0/docs/sdk.md),
+and [official 0.2.0 release](https://github.com/NVIDIA/NeMo-Speech.cpp/releases/tag/v0.2.0).

package/docs/reference/git-text-snapshot.md

@@ -0,0 +1,104 @@
+# Complete Git text snapshots
+
+`createGitTextSnapshot` from `arcane-os` reads application-selected UTF-8 text
+from one fetched Git revision. It uses a dedicated **bare object cache**, not a
+clone with a working checkout. It never reads uncommitted application files,
+checks out files, modifies a user checkout/index, or creates another worktree.
+Construction performs no I/O. The native host needs Git available on its PATH;
+Git owns remote access, authentication and its actual failures.
+
+```js
+import {createGitTextSnapshot} from 'arcane-os';
+
+const snapshot = createGitTextSnapshot({
+    cacheDirectory: applicationCacheDirectory,
+    remote: applicationRepository,
+    ref: applicationRef,
+    selectPath: function selectMoonDispatches(filename) {
+        return filename.startsWith('dispatches/') && filename.endsWith('.md');
+    }
+});
+
+const {revision, files} = await snapshot.refresh({signal});
+// files: [{path: 'dispatches/cheese-emergency.md', content: completeText}]
+await snapshot.close();
+```
+
+The application owns its remote, ref, path policy, repository identity, refresh
+schedule and Core service factory. The SDK does not select `main`, poll, parse
+Markdown/YAML, label or rewrite documents, or choose a product RPC method.
+`selectPath(path)` may return a boolean or a promise of one. It receives each
+complete repository-relative leaf path in Git tree order. Keep the selector's
+policy stable for the lifetime of its snapshot owner.
+
+## API and lifecycle
+
+`createGitTextSnapshot({cacheDirectory,remote,ref,selectPath,onEvent,run})`
+returns `{refresh,close,drain,dispose}`. `run` defaults to the SDK's existing
+`runProcess`; applications normally omit it. It is the same process adapter
+contract, not a second process supervisor. `onEvent` is the ordinary SDK event
+callback and follows its asynchronous completion/error contract.
+
+- `refresh({signal}={})` fetches the selected ref and returns
+  `{revision,files:[{path,content}]}`. `revision` is Git's opaque selected commit
+  name, not an SDK integrity record. Every file comes from that exact revision,
+  even if the remote moves during retrieval. Order follows Git's tree listing.
+- Concurrent calls share the active refresh. A later call fetches again; when
+  the selected revision is unchanged, it reuses the completed text retrieval.
+  Returned arrays and records are independent copies, so caller edits do not
+  alter retained results. Failed refreshes report errors, never stale success.
+- A caller's abort cancels only that caller's wait. Accepted shared refresh
+  continues, including when every caller leaves. There is no incomplete result
+  presented as a complete snapshot. A pre-aborted call starts no work.
+- `close()`, `drain()` and `dispose()` are the same idempotent operation. They
+  stop acceptance and await accepted refresh work and its process/event cleanup.
+  Calls after close reject with `CORE_CLOSING`. A failure during drain rejects
+  close. Connect this hook to the Core service's `drain`/`dispose` lifecycle;
+  do not terminate its process before accepted work settles.
+
+One application service owns each cache directory. Sharing that mutable bare
+cache across independent instances or processes requires coordination by its
+application owner; this factory does not install a daemon or cross-process lock.
+An empty cache is initialized once. An existing non-bare checkout is rejected
+without repurposing it. The cache is retained when the owner closes.
+
+## Complete text and observable failures
+
+Content is the exact Git blob text, including whitespace, CRLF, newlines,
+leading BOM and embedded NUL characters. No checkout filters, `textconv`,
+normalization, trimming, truncation, freezing or application size gate applies.
+Symlink blobs expose their stored target text; they are never followed.
+Selected submodules are not text blobs and produce an explicit error rather
+than being skipped or fetched as another repository. A selected path or blob
+that cannot be decoded as UTF-8 fails with `ARCANE_GIT_SNAPSHOT_NOT_TEXT` instead
+of silently substituting replacement characters. Git errors and complete
+diagnostics remain available to the caller and the existing process event owner.
+
+`git.snapshot.refreshing`, `git.snapshot.completed` and `git.snapshot.failed`
+describe the shared operation. Completion data contains `{revision,reused}`;
+failure data contains the actual error. These events supplement the complete
+result, never replace it. An event callback must not await a new refresh of the
+same owner from inside the operation it is currently observing.
+
+Initial refresh performs bare initialization/role discovery, then fetch,
+revision resolution, one tree listing and one batched blob process. Later
+unchanged revisions need only fetch and revision resolution. A changed revision
+uses one tree process and one blob process regardless of the selected file
+count. An empty selection needs no blob process. Blob requests use the shared
+async process-input owner; raw stdout is consumed by the Git parser with
+backpressure. Git's encoded body extent and object names remain local to its
+[batch protocol](https://git-scm.com/docs/git-cat-file#_batch_output); they do
+not become product metadata, byte progress, admission or integrity policy.
+
+## Core service composition
+
+The application wraps this mechanism in its own service factory and supplies its
+own public method and result fields. The handler can pass `context.signal` to
+`refresh` for request cancellation, then map `revision` to its chosen remote-ref
+field without changing any `content`. Return the snapshot owner's `drain` and
+`dispose` hooks from that service. Native launch configuration selects the
+cache location; browser code does not launch Git or reconstruct its protocol.
+
+This contract is portable Node source for Windows, Linux and macOS. Android
+requires a host that supplies the same Git/process capability; it is not a
+claim that every Android host ships Git.