@qvac/sdk 0.8.1 → 0.8.3

package/README.md

@@ -1,9 +1,3 @@
-[![QVAC logo](docs/logo.avif)](https://qvac.tether.dev)
-
----
-
-**QVAC** is an open-source, cross-platform **ecosystem** for local-first, peer-to-peer **AI**. QVAC runs on [**Bare**](https://bare.pears.com) by [Holepunch](https://holepunch.to), a lightweight, cross-platform JavaScript runtime.
-
 # QVAC SDK
 
 **QVAC SDK** is the canonical entry point to develop AI applications with QVAC.
@@ -11,140 +5,75 @@
 > _Part of **QVAC** ecosystem_
 > <br>
 > <sup>
-> <a href="https://qvac.tether.dev" >Home</a> &nbsp;•&nbsp;
-> <a href="https://qvac.tether.dev/docs" >Docs</a> &nbsp;•&nbsp;
-> <a href="https://qvacbytether.featurebase.app" >Support</a> &nbsp;•&nbsp;
+> <a href="https://qvac.tether.io/" >Home</a> &nbsp;•&nbsp;
+> <a href="https://docs.qvac.tether.io/" >Docs</a> &nbsp;•&nbsp;
+> <a href="https://discord.com/channels/1425125849346216029/1445400675189264516" >Support</a> &nbsp;•&nbsp;
 > <a href="https://discord.com/invite/tetherdev" >Discord</a>
-> </sup>
-
-Written in TypeScript, it provides all QVAC capabilities through a unified interface while also abstracting away the complexity of running your application in a JS environment other than Bare.
 
-For the comprehensive reference, [see official QVAC documentation](https://qvac.tether.dev/docs).
+**QVAC SDK** is the main entry point for developing applications with QVAC. It is type-safe and exposes all QVAC capabilities through a unified interface. It runs on Node.js, [Bare runtime](https://bare.pears.com), and [Expo](https://expo.dev).
 
-## Requirements
+See [https://docs.qvac.tether.io/sdk/getting-started](https://docs.qvac.tether.io/sdk/getting-started) for the comprehensive QVAC documentation.
 
-Supported JS environments: Bare, Node.js, Expo and Bun.
+For AI/LLM tools, use [https://docs.qvac.tether.io/llms-full.txt](https://docs.qvac.tether.io/llms-full.txt) as the consolidated plaintext documentation export.
 
-## Installation
+## Supported environments and installation
 
-```bash
-npm install @qvac/sdk
-```
+See https://docs.qvac.tether.io/sdk/getting-started/installation
 
-### Linux
+## Quickstart
 
-OS peer dependency:
+1. Create the examples workspace:
 
 ```bash
-apt install vulkan-sdk
+mkdir qvac-examples
+cd qvac-examples
+npm init -y && npm pkg set type=module
 ```
 
-### Expo
-
-1. Peer dependencies:
+2. Install the SDK:
 
 ```bash
-npm i expo-file-system react-native-bare-kit
-```
-
-2. On Android, bump `minSdkVersion` to 29, by adding `ext { minSdkVersion=29 }` to `android/build.gradle` or using `expo-build-properties`.
-
-3. Add the QVAC Expo plugin to `app.json`:
-
-```js
-export default {
-  expo: {
-    plugins: ["@qvac/sdk/expo-plugin"],
-  },
-};
-```
-
-4. Prebuild your project to generate the native files:
-
-```bash
-npx expo prebuild
-```
-
-5. Build and run it on a **physical device**:
-
-```bash
-npx expo run:ios --device
-# or
-npx expo run:android --device
+npm install @qvac/sdk
 ```
 
-> [!IMPORTANT]  
-> Due to limitations with `llamacpp`, QVAC currently does not run on emulators. You **must** use a physical device.
-
-## Usage
+3. Create the quickstart script:
 
 ```js
-import {
-  completion,
-  LLAMA_3_2_1B_INST_Q4_0,
-  loadModel,
-  downloadAsset,
-  unloadModel,
-  VERBOSITY,
-} from "@qvac/sdk";
+import { loadModel, LLAMA_3_2_1B_INST_Q4_0, completion, unloadModel, } from "@qvac/sdk";
 try {
-  // First just cache the model
-  await downloadAsset({
-    assetSrc: LLAMA_3_2_1B_INST_Q4_0,
-    onProgress: (progress) => {
-      console.log(progress);
-    },
-  });
-  // Then load it in memory from cache
-  const modelId = await loadModel({
-    modelSrc: LLAMA_3_2_1B_INST_Q4_0,
-    modelType: "llm",
-    modelConfig: {
-      device: "gpu",
-      ctx_size: 2048,
-      verbosity: VERBOSITY.ERROR,
-    },
-  });
-  const history = [
-    {
-      role: "user",
-      content: "Explain quantum computing in one sentence, use lots of emojis",
-    },
-  ];
-  const result = completion({ modelId, history, stream: true });
-  for await (const token of result.tokenStream) {
-    process.stdout.write(token);
-  }
-  const stats = await result.stats;
-  console.log("\n📊 Performance Stats:", stats);
-  // Change `clearStorage: true` to delete cached model files
-  await unloadModel({ modelId, clearStorage: false });
-} catch (error) {
-  console.error("❌ Error:", error);
-  process.exit(1);
+    // Load a model into memory
+    const modelId = await loadModel({
+        modelSrc: LLAMA_3_2_1B_INST_Q4_0,
+        modelType: "llm",
+        onProgress: (progress) => {
+            console.log(progress);
+        },
+    });
+    // You can use the loaded model multiple times
+    const history = [
+        {
+            role: "user",
+            content: "Explain quantum computing in one sentence",
+        },
+    ];
+    const result = completion({ modelId, history, stream: true });
+    for await (const token of result.tokenStream) {
+        process.stdout.write(token);
+    }
+    // Unload model to free up system resources
+    await unloadModel({ modelId });
+}
+catch (error) {
+    console.error("❌ Error:", error);
+    process.exit(1);
 }
 ```
 
-## Functionalities
-
-### AI tasks
-
-- Completion: LLM inference via [`llama.cpp`](https://github.com/ggml-org/llama.cpp).
-- Transcription: speech-to-text (ASR) via [`whisper.cpp`](https://github.com/ggml-org/whisper.cpp).
-- Text embeddings: via `llama.cpp`, for RAG.
-- Translation: between different languages.
-- Text-to-Speech: TTS via ONNX.
-- Multimodal: via [`llama.cpp`] — i.e., process and understand multiple types of media within the same conversation context.
-- RAG: retrieval-augmented generation with progress streaming, cancellation, and workspace management.
-- Delegated inference: perform peer-to-peer edge inference via Holepunch stack.
+4. Run the quickstart script:
 
-### Utilities
-
-- Configuration: customize SDK behavior via config files (`qvac.config.json`, `.js`, or `.ts`).
-- Logging: visibility into what's happening inside your models during loading, inference, and other operations.
-- Download Lifecycle: pause and resume model downloads.
-- Blind Relays: establish peer connections through NAT/firewalls by routing traffic through relay nodes.
-- Sharded models: download a model that is sharded into multiple parts.
+```bash
+node quickstart.js
+```
 
 ## Examples
 
@@ -164,161 +93,27 @@ node dist/examples/path/to/example.js
 bun run examples/path/to/example.ts
 ```
 
-### Completion
-
-- `llama.cpp` with local files: [`examples/llamacpp-filesystem.ts`](examples/llamacpp-filesystem.ts)
-- `llama.cpp` with P2P registry: [`examples/llamacpp-p2p.ts`](examples/llamacpp-p2p.ts)
-- `llama.cpp` with HTTP: [`examples/llamacpp-http.ts`](examples/llamacpp-http.ts)
-- `llama.cpp` with tools/function calls: [`examples/llamacpp-native-tools.ts`](examples/llamacpp-native-tools.ts)
-- `llama.cpp` with multimodal inference: [`examples/llamacpp-multimodal.ts`](examples/llamacpp-multimodal.ts)
-- `llama.cpp` with KV cache: [`examples/kv-cache-example.ts`](examples/kv-cache-example.ts)
-
-### Transcription
-
-- `whisper.cpp` transcription: [`examples/whispercpp-filesystem.ts`](examples/whispercpp-filesystem.ts)
-- Microphone recording: [`examples/whispercpp-microphone-record.ts`](examples/whispercpp-microphone-record.ts)
-
-### Embeddings
-
-- Single and batch embeddings: [`examples/embed-p2p.ts`](examples/embed-p2p.ts)
-
-**RAG with HyperDB** (cross-platform):
-
-- Ingest (full pipeline): [`examples/rag/rag-hyperdb/ingest.ts`](examples/rag/rag-hyperdb/ingest.ts)
-- Segregated pipeline: [`examples/rag/rag-hyperdb/pipeline.ts`](examples/rag/rag-hyperdb/pipeline.ts) _(Segregated flow: chunk → embed → save)_
-- Workspaces: [`examples/rag/rag-hyperdb/workspaces.ts`](examples/rag/rag-hyperdb/workspaces.ts) _(Workspace lifecycle: list, close, delete)_
-- Cancellation: [`examples/rag/rag-hyperdb/cancellation.ts`](examples/rag/rag-hyperdb/cancellation.ts) _(progress + cancel)_
-
-**RAG with other backends** (desktop only):
-
-- LanceDB: [`examples/rag/rag-lancedb.ts`](examples/rag/rag-lancedb.ts)
-- ChromaDB: [`examples/rag/rag-chromadb.ts`](examples/rag/rag-chromadb.ts) _(requires ChromaDB server)_
-- SQLite-Vector: [`examples/rag/rag-sqlite.ts`](examples/rag/rag-sqlite.ts) _(SQLite-Vector WASM)_
-
-### Translation
-
-- Marian OPUS translation: [`examples/translation/translation-opus.ts`](examples/translation/translation-opus.ts)
-- Indic language translation: [`examples/translation/translation-indic.ts`](examples/translation/translation-indic.ts)
-- LLM-based translation: [`examples/translation/translation-llm.ts`](examples/translation/translation-llm.ts)
-
-### Text-to-Speech
-
-- TTS (Chatterbox): [`examples/tts/chatterbox.ts`](examples/tts/chatterbox.ts) _(voice cloning with reference audio)_
-- TTS (Supertonic): [`examples/tts/supertonic.ts`](examples/tts/supertonic.ts) _(general-purpose, no voice cloning)_
-
-### Multimodel
-
-- Load multiple models simultaneously: [`examples/multi-model-demo.ts`](examples/multi-model-demo.ts)
-
-### Delegated inference
-
-- Provider: [`examples/delegated-inference/provider.ts`](examples/delegated-inference/provider.ts)
-- Consumer: [`examples/delegated-inference/consumer.ts`](examples/delegated-inference/consumer.ts)
-
-> [!TIP]
-> Set `QVAC_HYPERSWARM_SEED` env var to ensure that the provider uses the same keypair (i.e., public key doesn't change on every run).
-
-> [!NOTE]
-> Consumer does not handle reconnection yet.
-
-### Logging
-
-Stream real-time logs from the SDK server and native addons:
-
-```ts
-import { loggingStream, SDK_LOG_ID } from "@qvac/sdk";
+## Build
 
-// SDK server logs (general operations)
-for await (const log of loggingStream({ id: SDK_LOG_ID })) {
-  console.log(`[${log.level}] ${log.namespace}: ${log.message}`);
-}
+Use the [Bun](https://bun.sh/) package manager:
 
-// Addon logs per model (llamacpp, whispercpp, etc.)
-for await (const log of loggingStream({ id: modelId })) {
-  console.log(`[${log.level}] ${log.namespace}: ${log.message}`);
-}
+```bash
+bun i
 ```
 
-- Log streaming: [`examples/logging-streaming.ts`](examples/logging-streaming.ts)
-- Log with custom file transport: [`examples/logging-file-transport.ts`](examples/logging-file-transport.ts)
-
-### Configuration
-
-Customize SDK behavior using a config file. The SDK auto-discovers `qvac.config.{json,js,ts}` in your project root, or you can specify a path via `QVAC_CONFIG_PATH` environment variable.
-
-**Supported formats:**
-
-- `qvac.config.json` - JSON format
-- `qvac.config.js` - JavaScript with `export default`
-- `qvac.config.ts` - TypeScript with `export default`
-
-**Available options:**
-
-| Option                    | Type       | Default          | Description                                         |
-| ------------------------- | ---------- | ---------------- | --------------------------------------------------- |
-| `cacheDirectory`          | `string`   | `~/.qvac/models` | Where models and assets are stored                  |
-| `swarmRelays`             | `string[]` | `[]`             | Hyperswarm relay public keys for P2P                |
-| `loggerLevel`             | `string`   | `"info"`         | Log level: `"error"`, `"warn"`, `"info"`, `"debug"` |
-| `loggerConsoleOutput`     | `boolean`  | `true`           | Enable/disable console output                       |
-| `httpDownloadConcurrency` | `number`   | `3`              | Max concurrent HTTP downloads for sharded models    |
-| `httpConnectionTimeoutMs` | `number`   | `10000`          | HTTP connection timeout in milliseconds             |
-
-- Config usage example: [`examples/default-config-usage.ts`](examples/default-config-usage.ts)
-
-### Download Lifecycle
-
-- Pause and resume download: [`examples/download-with-cancel.ts`](examples/download-with-cancel.ts)
-
-### Blind Relays
-
-Blind relays help establish peer connections through NAT/firewalls by routing traffic through relay nodes.
-
-- Model downloads via Hyperdrive: [`examples/download-with-blind-relays.ts`](./examples/download-with-blind-relays.ts)
-- Delegated inference: You can reuse the same pattern for delegated inference by adding `swarmRelays` to your config file before starting your provider/consumer.
-
-> [!NOTE]
-> The examples use mock relay keys. For real deployments, you **must** use your own relay servers or trusted public relays.
-
-### Sharded Models
-
-Sharded models are split into multiple files following the pattern: `<name>-00001-of-0000X.<ext>`. The SDK automatically downloads and loads all parts with detailed progress tracking.
-
-**Supported formats:**
-
-- Archives (`.tar`, `.tar.gz`, `.tgz`): HTTP or local with automatic extraction
-- HTTP sharded URL: pass the download URL of any shard and the SDK will fetch the remaining shards
-- Hyperdrive: use any sharded Hyperdrive model source
-- Local shards: pass the path to any shard file. _(Note: All shards must be in the same directory)_
-
-See: [`examples/llamacpp-sharded.ts`](examples/llamacpp-sharded.ts)
-
-## Basic flow
-
-```mermaid
-sequenceDiagram
-    participant User as User
-    participant SDK as QVAC SDK
-    participant RPC as RPC Client (singleton)
-    participant Worker as Bare Worker (singleton)
-
-    User->>SDK: Call loadModel("llama-3", options)
-    SDK->>RPC: Create runtime-specific RPC client
-    RPC->>Worker: Spawn worker (if needed) and send loadModel request
-    Worker->>Worker: Download and load model into memory
-    Worker->>RPC: Return success response
-    RPC->>SDK: Model loaded successfully
-    SDK->>User: loadModel() resolves with modelId
+```bash
+bun run build  # or `watch` for hotreload
 ```
 
-**Note**: The example uses mock relay keys. In real deployments, you **must** use your own relay servers or trusted public relays.
-
-### Hot Config Reload
-
-Hot config reload allows you to update model configurations on-the-fly without unloading the model. Pass `modelId` (instead of `modelSrc`) to `loadModel` with the `modelType` and new `modelConfig` to apply changes instantly.
+```bash
+bun run build:pack
+```
 
-- Config reload using whisper: [`examples/config-reload.ts`](examples/config-reload.ts)
+This outputs a tarball under `dist/sdk-{version}.tgz` that you can install in your project, e.g.:
 
-**Note**: Config reload is currently supported for Whisper models. All config parameters except `contextParams` (GPU settings, flash attention) can be hot reloaded. `contextParams` are load-time only and require full model reload. More model types coming soon.
+```bash
+npm i path/to/sdk-0.3.0.tgz
+```
 
 ## Contributing
 
@@ -459,32 +254,3 @@ This will:
 4. Generate `changelog/<version>/CHANGELOG.md`
 5. Generate `changelog/<version>/breaking.md` for BC changes (with code examples)
 6. Generate `changelog/<version>/api.md` for API changes (with code examples)
-
-**Note:** Requires a GitHub token (`GITHUB_TOKEN` or `GH_TOKEN` environment variable) to fetch PR metadata.
-
-## Build
-
-Use the [Bun](https://bun.sh/) package manager:
-
-```bash
-bun i
-```
-
-```bash
-bun run build  # or `watch` for hotreload
-```
-
-```bash
-bun run build:pack
-```
-
-This outputs a tarball under `dist/sdk-{version}.tgz` that you can install in your project, e.g.:
-
-```bash
-npm i path/to/sdk-0.3.0.tgz
-```
-
-## More Resources
-
-- [Comprehensive documentation of this SDK](https://qvac.tether.dev/docs/sdk)
-- [Package at NPM](https://www.npmjs.com/package/@qvac/sdk)

package/dist/server/bare/plugins/llamacpp-completion/ops/cache-logger.d.ts

@@ -3,4 +3,5 @@ export declare function logCacheInit(cacheKey: string, systemPrompt: string, too
 export declare function logMessagesToAddon(messages: unknown[], phase?: "CACHE_INIT" | "PROMPT_SEND" | "NO_CACHE"): void;
 export declare function logCacheDisabled(): void;
 export declare function logCacheSave(sessionPath: string): void;
+export declare function logCacheSaveError(sessionPath: string, err: unknown): void;
 //# sourceMappingURL=cache-logger.d.ts.map

package/dist/server/bare/plugins/llamacpp-completion/ops/cache-logger.js

@@ -32,3 +32,6 @@ export function logCacheDisabled() {
 export function logCacheSave(sessionPath) {
     logger.debug(`[kv-cache] Saving session: ...${sessionPath.slice(-20)}`);
 }
+export function logCacheSaveError(sessionPath, err) {
+    logger.warn(`[kv-cache] Failed to save session: ...${sessionPath.slice(-20)} — ${err instanceof Error ? err.message : String(err)}`);
+}

package/dist/server/bare/plugins/llamacpp-completion/ops/completion-stream.js

@@ -1,4 +1,4 @@
-import { logCacheDisabled, logCacheInit, logCacheSave, logCacheStatus, logMessagesToAddon, } from "./cache-logger.js";
+import { logCacheDisabled, logCacheInit, logCacheSave, logCacheSaveError, logCacheStatus, logMessagesToAddon, } from "./cache-logger.js";
 import { customCacheExists, extractSystemPrompt, findMatchingCache, generateConfigHash, getCacheFilePath, getCurrentCacheInfo, markCacheInitialized, renameCacheFile, } from "../../../ops/kv-cache-utils.js";
 import { getModel, getModelConfig, } from "../../../registry/model-registry.js";
 import { checkForToolEvents, insertToolsIntoHistory, setupToolGrammar, } from "../../../../utils/tool-integration.js";
@@ -104,8 +104,18 @@ async function* processModelResponse(model, messagesToSend, shouldSaveCache, too
         const sessionMsg = messagesToSend.find((m) => m.role === "session");
         if (sessionMsg?.content) {
             logCacheSave(sessionMsg.content);
+            const cachePath = sessionMsg.content;
+            const saveResp = await model.run([
+                { role: "session", content: cachePath },
+                { role: "session", content: "save" },
+            ]);
+            try {
+                await saveResp.await();
+            }
+            catch (err) {
+                logCacheSaveError(cachePath, err);
+            }
         }
-        await model.run([{ role: "session", content: "save" }]);
     }
     const responseWithStats = response;
     const stats = {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@qvac/sdk",
-  "version": "0.8.1",
+  "version": "0.8.3",
   "license": "Apache-2.0",
   "repository": {
     "type": "git",
@@ -9,6 +9,51 @@
   },
   "bugs": "https://github.com/tetherto/qvac/issues",
   "homepage": "https://github.com/tetherto/qvac/tree/main/packages/sdk#readme",
+  "keywords": [
+    "ai",
+    "bare",
+    "cross-platform",
+    "deep-learning",
+    "diffusion",
+    "edge-ai",
+    "embeddings",
+    "expo",
+    "gguf",
+    "gpu",
+    "holepunch",
+    "hyperswarm",
+    "image-generation",
+    "inference",
+    "large-language-model",
+    "llama-cpp",
+    "llm",
+    "local",
+    "local-ai",
+    "machine-learning",
+    "ml",
+    "model-registry",
+    "node",
+    "ocr",
+    "offline",
+    "on-device",
+    "p2p",
+    "peer-to-peer",
+    "private-ai",
+    "qvac",
+    "rag",
+    "react-native",
+    "sdk",
+    "self-hosted",
+    "speech-to-text",
+    "stt",
+    "tether",
+    "text-to-speech",
+    "transcription",
+    "translation",
+    "tts",
+    "typescript",
+    "whisper"
+  ],
   "type": "module",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",