@qvac/sdk 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +328 -200
- package/dist/client/api/completion-stream.d.ts +15 -5
- package/dist/client/api/completion-stream.js +50 -14
- package/dist/client/api/embed.d.ts +17 -3
- package/dist/client/api/embed.js +0 -8
- package/dist/client/api/get-model-info.d.ts +1 -1
- package/dist/client/api/index.d.ts +0 -1
- package/dist/client/api/index.js +0 -1
- package/dist/client/api/load-model.d.ts +37 -11
- package/dist/client/api/load-model.js +29 -117
- package/dist/client/api/logging-stream.d.ts +6 -3
- package/dist/client/api/logging-stream.js +6 -3
- package/dist/client/api/unload-model.js +5 -2
- package/dist/client/config-loader/config-utils.d.ts +5 -0
- package/dist/client/config-loader/config-utils.js +20 -0
- package/dist/client/config-loader/resolve-config.bare.d.ts +9 -0
- package/dist/client/config-loader/resolve-config.bare.js +87 -0
- package/dist/client/config-loader/resolve-config.expo.d.ts +11 -0
- package/dist/client/config-loader/resolve-config.expo.js +62 -0
- package/dist/client/config-loader/resolve-config.node.d.ts +9 -0
- package/dist/client/config-loader/resolve-config.node.js +118 -0
- package/dist/client/init-hooks.d.ts +17 -3
- package/dist/client/init-hooks.js +37 -18
- package/dist/client/logging-stream-registry.d.ts +1 -1
- package/dist/client/logging-stream-registry.js +14 -13
- package/dist/client/rpc/bare-client.d.ts +10 -14
- package/dist/client/rpc/bare-client.js +28 -2
- package/dist/client/rpc/bare-rpc-client.d.ts +1 -1
- package/dist/client/rpc/bare-rpc-client.js +16 -13
- package/dist/client/rpc/expo-rpc-client.d.ts +2 -2
- package/dist/client/rpc/expo-rpc-client.js +15 -12
- package/dist/client/rpc/node-rpc-client.d.ts +1 -1
- package/dist/client/rpc/node-rpc-client.js +10 -7
- package/dist/client/rpc/rpc-client.js +16 -15
- package/dist/constants/audio.d.ts +1 -2
- package/dist/constants/audio.js +1 -17
- package/dist/examples/config/config-path.d.ts +8 -0
- package/dist/examples/config/config-path.js +13 -0
- package/dist/examples/config-reload.d.ts +2 -0
- package/dist/examples/config-reload.js +49 -0
- package/dist/examples/download-with-blind-relays.js +20 -20
- package/dist/examples/embed-hyperdrive.js +35 -3
- package/dist/examples/kv-cache-cleanup.js +1 -1
- package/dist/examples/kv-cache-custom-key.js +1 -1
- package/dist/examples/kv-cache-example.js +1 -1
- package/dist/examples/llamacpp-hyperdrive.js +1 -1
- package/dist/examples/llamacpp-sharded.js +1 -1
- package/dist/examples/logging-file-transport.d.ts +2 -0
- package/dist/examples/{llamacpp-file-logging.js → logging-file-transport.js} +3 -7
- package/dist/examples/logging-streaming.d.ts +2 -0
- package/dist/examples/logging-streaming.js +75 -0
- package/dist/examples/mcp-websearch.d.ts +13 -0
- package/dist/examples/mcp-websearch.js +176 -0
- package/dist/index.d.ts +4 -4
- package/dist/index.js +2 -2
- package/dist/logging/addon.d.ts +6 -0
- package/dist/logging/addon.js +59 -0
- package/dist/logging/base-logger.js +24 -14
- package/dist/logging/client-logger.d.ts +3 -0
- package/dist/logging/client-logger.js +13 -0
- package/dist/logging/index.d.ts +6 -1
- package/dist/logging/index.js +6 -1
- package/dist/logging/logger.d.ts +0 -2
- package/dist/logging/logger.js +0 -3
- package/dist/logging/namespaces.d.ts +11 -0
- package/dist/logging/namespaces.js +11 -0
- package/dist/logging/registry.d.ts +26 -0
- package/dist/logging/registry.js +36 -0
- package/dist/logging/server-logger.d.ts +3 -0
- package/dist/logging/server-logger.js +16 -0
- package/dist/logging/stream-logger.d.ts +1 -1
- package/dist/logging/stream-logger.js +6 -3
- package/dist/logging/types.d.ts +2 -0
- package/dist/models/hyperdrive/models.d.ts +789 -51
- package/dist/models/hyperdrive/models.js +979 -200
- package/dist/schemas/common.d.ts +86 -23
- package/dist/schemas/common.js +0 -3
- package/dist/schemas/download-asset.d.ts +5 -5
- package/dist/schemas/embed.d.ts +3 -3
- package/dist/schemas/embed.js +9 -2
- package/dist/schemas/get-model-info.d.ts +2 -2
- package/dist/schemas/index.d.ts +5 -1
- package/dist/schemas/index.js +5 -1
- package/dist/schemas/json-schema.d.ts +16 -0
- package/dist/schemas/json-schema.js +2 -0
- package/dist/schemas/llamacpp-config.d.ts +5 -4
- package/dist/schemas/llamacpp-config.js +7 -5
- package/dist/schemas/load-model.d.ts +931 -211
- package/dist/schemas/load-model.js +7 -1
- package/dist/schemas/logging-stream.d.ts +10 -3
- package/dist/schemas/logging-stream.js +4 -3
- package/dist/schemas/mcp-adapter.d.ts +47 -0
- package/dist/schemas/mcp-adapter.js +1 -0
- package/dist/schemas/model-src-utils.d.ts +8 -6
- package/dist/schemas/model-src-utils.js +8 -3
- package/dist/schemas/reload-config.d.ts +344 -0
- package/dist/schemas/reload-config.js +42 -0
- package/dist/schemas/sdk-config.d.ts +22 -0
- package/dist/schemas/sdk-config.js +33 -0
- package/dist/schemas/sdk-errors-client.d.ts +4 -0
- package/dist/schemas/sdk-errors-client.js +20 -0
- package/dist/schemas/sdk-errors-server.d.ts +3 -0
- package/dist/schemas/sdk-errors-server.js +15 -0
- package/dist/schemas/tools.d.ts +3 -0
- package/dist/schemas/translation-config.d.ts +81 -0
- package/dist/schemas/translation-config.js +3 -4
- package/dist/schemas/whispercpp-config.d.ts +9 -1
- package/dist/schemas/whispercpp-config.js +2 -1
- package/dist/server/bare/addons/llamacpp/create-model.js +13 -4
- package/dist/server/bare/addons/llamacpp/handlers/embed.d.ts +9 -1
- package/dist/server/bare/addons/llamacpp/handlers/embed.js +18 -4
- package/dist/server/bare/addons/llamacpp/kv-cache-utils.js +6 -4
- package/dist/server/bare/addons/load-model.js +10 -4
- package/dist/server/bare/addons/unload-model.js +6 -3
- package/dist/server/bare/addons/whispercpp/handlers/transcribe-stream.js +56 -37
- package/dist/server/bare/delegate-rpc-client.js +12 -10
- package/dist/server/bare/hyperswarm.js +7 -5
- package/dist/server/bare/rag-hyperdb/rag-manager.js +3 -1
- package/dist/server/bare/registry/config-registry.d.ts +13 -4
- package/dist/server/bare/registry/config-registry.js +42 -10
- package/dist/server/bare/registry/logging-stream-registry.d.ts +24 -7
- package/dist/server/bare/registry/logging-stream-registry.js +99 -20
- package/dist/server/bare/registry/model-registry.d.ts +3 -0
- package/dist/server/bare/registry/model-registry.js +9 -6
- package/dist/server/rpc/create-server.js +5 -3
- package/dist/server/rpc/handle-request.js +30 -16
- package/dist/server/rpc/handlers/cancelHandler.js +3 -1
- package/dist/server/rpc/handlers/completion-stream-delegated.js +4 -2
- package/dist/server/rpc/handlers/delete-cache.js +3 -1
- package/dist/server/rpc/handlers/download-asset.js +3 -1
- package/dist/server/rpc/handlers/embed.js +3 -1
- package/dist/server/rpc/handlers/index.d.ts +0 -1
- package/dist/server/rpc/handlers/index.js +0 -2
- package/dist/server/rpc/handlers/load-model/download-manager.js +4 -2
- package/dist/server/rpc/handlers/load-model/http.js +20 -18
- package/dist/server/rpc/handlers/load-model/hyperdrive.js +21 -19
- package/dist/server/rpc/handlers/load-model/resolve.js +7 -5
- package/dist/server/rpc/handlers/load-model-delegated.d.ts +2 -2
- package/dist/server/rpc/handlers/load-model-delegated.js +10 -8
- package/dist/server/rpc/handlers/load-model.js +55 -3
- package/dist/server/rpc/handlers/logging-stream.js +4 -4
- package/dist/server/rpc/handlers/provideHandler/connection.js +12 -10
- package/dist/server/rpc/handlers/provideHandler/handler.js +18 -16
- package/dist/server/rpc/handlers/provideHandler/proxy.js +8 -6
- package/dist/server/rpc/handlers/rag.js +3 -1
- package/dist/server/rpc/handlers/stopProvideHandler.js +3 -1
- package/dist/server/rpc/handlers/transcribe-stream.js +3 -1
- package/dist/server/rpc/handlers/translate.js +3 -1
- package/dist/server/rpc/handlers/unload-model.js +4 -2
- package/dist/server/utils/audio/decoder.d.ts +2 -1
- package/dist/server/utils/audio/decoder.js +49 -12
- package/dist/server/utils/cache.js +7 -5
- package/dist/server/utils/config-transform.d.ts +3 -0
- package/dist/server/utils/config-transform.js +21 -0
- package/dist/server/utils/index.d.ts +1 -0
- package/dist/server/utils/index.js +1 -0
- package/dist/server/worker.js +41 -16
- package/dist/utils/errors-client.d.ts +12 -0
- package/dist/utils/errors-client.js +20 -0
- package/dist/utils/errors-server.d.ts +9 -0
- package/dist/utils/errors-server.js +16 -0
- package/dist/utils/getProcessSafe.d.ts +7 -0
- package/dist/utils/getProcessSafe.js +17 -0
- package/dist/utils/mcp-adapter.d.ts +12 -0
- package/dist/utils/mcp-adapter.js +103 -0
- package/dist/utils/object.d.ts +2 -0
- package/dist/utils/object.js +3 -0
- package/dist/utils/runtime.d.ts +30 -0
- package/dist/utils/runtime.js +48 -0
- package/dist/utils/tool-helpers.d.ts +10 -2
- package/dist/utils/tool-helpers.js +24 -11
- package/package.json +47 -34
- package/dist/client/api/set-config.d.ts +0 -7
- package/dist/client/api/set-config.js +0 -20
- package/dist/client/config-manager.d.ts +0 -4
- package/dist/client/config-manager.js +0 -8
- package/dist/examples/llamacpp-file-logging.d.ts +0 -2
- package/dist/schemas/set-config.d.ts +0 -21
- package/dist/schemas/set-config.js +0 -14
- package/dist/server/rpc/handlers/set-config.d.ts +0 -3
- package/dist/server/rpc/handlers/set-config.js +0 -18
package/README.md
CHANGED
|
@@ -1,14 +1,25 @@
|
|
|
1
|
-
[](https://qvac.tether.dev)
|
|
2
2
|
|
|
3
3
|
---
|
|
4
4
|
|
|
5
|
-
**QVAC** is an open-source, cross-platform
|
|
5
|
+
**QVAC** is an open-source, cross-platform **ecosystem** for local-first, peer-to-peer **AI**. QVAC runs on [**Bare**](https://bare.pears.com) by [Holepunch](https://holepunch.to), a lightweight, cross-platform JavaScript runtime.
|
|
6
6
|
|
|
7
7
|
# QVAC SDK
|
|
8
8
|
|
|
9
|
-
**QVAC SDK** is the canonical entry point to
|
|
9
|
+
**QVAC SDK** is the canonical entry point to develop AI applications with QVAC.
|
|
10
10
|
|
|
11
|
-
|
|
11
|
+
> _Part of **QVAC** ecosystem_
|
|
12
|
+
> <br>
|
|
13
|
+
> <sup>
|
|
14
|
+
> <a href="https://qvac.tether.dev" >Home</a> •
|
|
15
|
+
> <a href="https://qvac.tether.dev/docs" >Docs</a> •
|
|
16
|
+
> <a href="https://qvacbytether.featurebase.app" >Support</a> •
|
|
17
|
+
> <a href="https://discord.com/invite/tetherdev" >Discord</a>
|
|
18
|
+
> </sup>
|
|
19
|
+
|
|
20
|
+
Written in TypeScript, it provides all QVAC capabilities through a unified interface while also abstracting away the complexity of running your application in a JS environment other than Bare.
|
|
21
|
+
|
|
22
|
+
For the comprehensive reference, [see official QVAC documentation](https://qvac.tether.dev/docs).
|
|
12
23
|
|
|
13
24
|
## Requirements
|
|
14
25
|
|
|
@@ -16,65 +27,29 @@ Supported JS environments: Bare, Node.js, Expo and Bun.
|
|
|
16
27
|
|
|
17
28
|
## Installation
|
|
18
29
|
|
|
19
|
-
### Steps
|
|
20
|
-
|
|
21
|
-
1. [Authenticate to Tether’s GitHub packages](#step-1-authenticate-to-tethers-github-packages)
|
|
22
|
-
2. [Install package and dependencies](#step-2-install-package-and-dependencies)
|
|
23
|
-
|
|
24
|
-
### Step 1: authenticate to Tether’s GitHub packages
|
|
25
|
-
|
|
26
|
-
1. [Create a GitHub personal access token (classic) with scope `read:packages`](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic).
|
|
27
|
-
|
|
28
|
-
2. Add the token to your environment variables. For example, on macOS/Linux:
|
|
29
|
-
|
|
30
30
|
```bash
|
|
31
|
-
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
3. In the root of your project, create file `.npmrc` with the following content:
|
|
35
|
-
|
|
36
|
-
```ini
|
|
37
|
-
@qvac:registry=https://registry.npmjs.org
|
|
38
|
-
//registry.npmjs.org/:_authToken=${NPM_TOKEN}
|
|
31
|
+
npm install @qvac/sdk
|
|
39
32
|
```
|
|
40
33
|
|
|
41
|
-
|
|
34
|
+
### Linux
|
|
42
35
|
|
|
43
|
-
|
|
36
|
+
OS peer dependency:
|
|
44
37
|
|
|
45
38
|
```bash
|
|
46
|
-
|
|
39
|
+
apt install vulkan-sdk
|
|
47
40
|
```
|
|
48
41
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
### Step 2: Install package and dependencies
|
|
42
|
+
### Expo
|
|
52
43
|
|
|
53
|
-
1.
|
|
54
|
-
|
|
55
|
-
```bash
|
|
56
|
-
npm i @qvac/sdk
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
> [!TIP]
|
|
60
|
-
> If you can't get the package via Tether's private GitHub package registry, see section [Build from source](#build-from-source).
|
|
61
|
-
|
|
62
|
-
#### On Linux
|
|
63
|
-
|
|
64
|
-
> [!IMPORTANT]
|
|
65
|
-
> If you're on Linux ensure that the Vulkan SDK is installed (e.g. `apt install vulkan-sdk` on Ubuntu)
|
|
66
|
-
|
|
67
|
-
#### On Expo
|
|
68
|
-
|
|
69
|
-
1. Install the peer dependencies for Expo/React Native support:
|
|
44
|
+
1. Peer dependencies:
|
|
70
45
|
|
|
71
46
|
```bash
|
|
72
47
|
npm i expo-file-system react-native-bare-kit
|
|
73
48
|
```
|
|
74
49
|
|
|
75
|
-
2.
|
|
50
|
+
2. On Android, bump `minSdkVersion` to 29, by adding `ext { minSdkVersion=29 }` to `android/build.gradle` or using `expo-build-properties`.
|
|
76
51
|
|
|
77
|
-
3. Add the QVAC Expo plugin to
|
|
52
|
+
3. Add the QVAC Expo plugin to `app.json`:
|
|
78
53
|
|
|
79
54
|
```js
|
|
80
55
|
export default {
|
|
@@ -101,78 +76,80 @@ npx expo run:android --device
|
|
|
101
76
|
> [!IMPORTANT]
|
|
102
77
|
> Due to limitations with `llamacpp`, QVAC currently does not run on emulators. You **must** use a physical device.
|
|
103
78
|
|
|
104
|
-
##
|
|
105
|
-
|
|
106
|
-
You can run the self-contained example below:
|
|
79
|
+
## Usage
|
|
107
80
|
|
|
108
81
|
```js
|
|
109
|
-
import {
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
82
|
+
import {
|
|
83
|
+
completion,
|
|
84
|
+
LLAMA_3_2_1B_INST_Q4_0,
|
|
85
|
+
loadModel,
|
|
86
|
+
downloadAsset,
|
|
87
|
+
unloadModel,
|
|
88
|
+
VERBOSITY,
|
|
89
|
+
} from "@qvac/sdk";
|
|
90
|
+
try {
|
|
91
|
+
// First just cache the model
|
|
92
|
+
await downloadAsset({
|
|
93
|
+
assetSrc: LLAMA_3_2_1B_INST_Q4_0,
|
|
94
|
+
onProgress: (progress) => {
|
|
95
|
+
console.log(progress);
|
|
96
|
+
},
|
|
97
|
+
});
|
|
98
|
+
// Then load it in memory from cache
|
|
99
|
+
const modelId = await loadModel({
|
|
100
|
+
modelSrc: LLAMA_3_2_1B_INST_Q4_0,
|
|
115
101
|
modelType: "llm",
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
history
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
const
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
// You can use the loaded model multiple times
|
|
141
|
-
|
|
142
|
-
// Unload model to free up system resources
|
|
143
|
-
await unloadModel({ modelId });
|
|
144
|
-
process.exit();
|
|
102
|
+
modelConfig: {
|
|
103
|
+
device: "gpu",
|
|
104
|
+
ctx_size: 2048,
|
|
105
|
+
verbosity: VERBOSITY.ERROR,
|
|
106
|
+
},
|
|
107
|
+
});
|
|
108
|
+
const history = [
|
|
109
|
+
{
|
|
110
|
+
role: "user",
|
|
111
|
+
content: "Explain quantum computing in one sentence, use lots of emojis",
|
|
112
|
+
},
|
|
113
|
+
];
|
|
114
|
+
const result = completion({ modelId, history, stream: true });
|
|
115
|
+
for await (const token of result.tokenStream) {
|
|
116
|
+
process.stdout.write(token);
|
|
117
|
+
}
|
|
118
|
+
const stats = await result.stats;
|
|
119
|
+
console.log("\n📊 Performance Stats:", stats);
|
|
120
|
+
// Change `clearStorage: true` to delete cached model files
|
|
121
|
+
await unloadModel({ modelId, clearStorage: false });
|
|
122
|
+
} catch (error) {
|
|
123
|
+
console.error("❌ Error:", error);
|
|
124
|
+
process.exit(1);
|
|
125
|
+
}
|
|
145
126
|
```
|
|
146
127
|
|
|
147
|
-
|
|
128
|
+
## Functionalities
|
|
148
129
|
|
|
149
|
-
|
|
130
|
+
### AI tasks
|
|
150
131
|
|
|
151
|
-
|
|
132
|
+
- Completion: LLM inference via [`llama.cpp`](https://github.com/ggml-org/llama.cpp).
|
|
133
|
+
- Transcription: speech-to-text (ASR) via [`whisper.cpp`](https://github.com/ggml-org/whisper.cpp).
|
|
134
|
+
- Text embeddings: via `llama.cpp`, for RAG.
|
|
135
|
+
- Translation: between different languages.
|
|
136
|
+
- Text-to-Speech: TTS via ONNX.
|
|
137
|
+
- Multimodal: via [`llama.cpp`] — i.e., process and understand multiple types of media within the same conversation context.
|
|
138
|
+
- RAG: out-of-the-box retrieval-augmented generation workflow.
|
|
139
|
+
- Delegated inference: perform peer-to-peer edge inference via Holepunch stack.
|
|
152
140
|
|
|
153
|
-
|
|
154
|
-
bun i
|
|
155
|
-
```
|
|
141
|
+
### Utilities
|
|
156
142
|
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
```bash
|
|
162
|
-
bun run build:pack
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
This outputs a tarball under `dist/qvac-sdk-{version}.tgz` that you can install in your project, e.g.:
|
|
166
|
-
|
|
167
|
-
```bash
|
|
168
|
-
npm i path/to/qvac-sdk-0.3.0.tgz
|
|
169
|
-
```
|
|
143
|
+
- Logging: visibility into what's happening inside your models during loading, inference, and other operations.
|
|
144
|
+
- Download Lifecycle: pause and resume model downloads.
|
|
145
|
+
- Blind Relays: establish peer connections through NAT/firewalls by routing traffic through relay nodes.
|
|
146
|
+
- Sharded models: download a model that is sharded into multiple parts.
|
|
170
147
|
|
|
171
148
|
## Examples
|
|
172
149
|
|
|
173
|
-
In the `./examples` subdirectory, you will find scripts demonstrating SDK
|
|
150
|
+
In the `./examples` subdirectory, you will find scripts demonstrating how to use all SDK functionalities. To try any of them:
|
|
174
151
|
|
|
175
|
-
1.
|
|
152
|
+
1. Build the SDK from source (see [Build](#build) section).
|
|
176
153
|
2. Run using Bare, Node.js, or Bun as the runtime:
|
|
177
154
|
|
|
178
155
|
```bash
|
|
@@ -186,97 +163,38 @@ node dist/examples/path/to/example.js
|
|
|
186
163
|
bun run examples/path/to/example.ts
|
|
187
164
|
```
|
|
188
165
|
|
|
189
|
-
## Logging
|
|
190
|
-
|
|
191
|
-
QVAC SDK includes built-in log propagation that streams model logs from the worker process to your client application in real-time. This gives you visibility into what's happening inside your models during loading, inference, and other operations.
|
|
192
|
-
|
|
193
|
-
### Usage
|
|
194
|
-
|
|
195
|
-
Simply pass a logger when loading your model:
|
|
196
|
-
|
|
197
|
-
```js
|
|
198
|
-
import { getLogger, loadModel, completion, unloadModel } from "@qvac/sdk";
|
|
199
|
-
|
|
200
|
-
// Create a logger for your application
|
|
201
|
-
const logger = getLogger("my-app", {
|
|
202
|
-
level: "debug", // Set log level
|
|
203
|
-
transports: [
|
|
204
|
-
// Optional: add custom log handlers
|
|
205
|
-
(level, namespace, message) => {
|
|
206
|
-
console.log(`[${level.toUpperCase()}] ${namespace}: ${message}`);
|
|
207
|
-
},
|
|
208
|
-
],
|
|
209
|
-
});
|
|
210
|
-
|
|
211
|
-
// Load model with logging - RPC stream starts automatically
|
|
212
|
-
const modelId = await loadModel(
|
|
213
|
-
"pear://afa79ee07c0a138bb9f11bfaee771fb1bdfca8c82d961cff0474e49827bd1de3/Llama-3.2-1B-Instruct-Q4_0.gguf",
|
|
214
|
-
{
|
|
215
|
-
modelType: "llm",
|
|
216
|
-
logger, // Enable log streaming
|
|
217
|
-
},
|
|
218
|
-
);
|
|
219
|
-
|
|
220
|
-
// Now you'll see model logs in your console as operations happen
|
|
221
|
-
for await (const token of completion({
|
|
222
|
-
modelId,
|
|
223
|
-
history: [{ role: "user", content: "Hello!" }],
|
|
224
|
-
}).tokenStream) {
|
|
225
|
-
process.stdout.write(token);
|
|
226
|
-
}
|
|
227
|
-
|
|
228
|
-
await unloadModel({ modelId }); // Logging stream closes automatically
|
|
229
|
-
```
|
|
230
|
-
|
|
231
|
-
### What You'll See
|
|
232
|
-
|
|
233
|
-
When logging is enabled, you'll see real-time logs from the underlying model libraries:
|
|
234
|
-
|
|
235
|
-
```
|
|
236
|
-
[DEBUG] llamacpp:llm: Loading model weights...
|
|
237
|
-
[INFO] llamacpp:llm: Model loaded successfully, vocab_size=32000
|
|
238
|
-
[DEBUG] llamacpp:llm: Starting inference...
|
|
239
|
-
[DEBUG] llamacpp:llm: Inference completed, tokens=12
|
|
240
|
-
```
|
|
241
|
-
|
|
242
|
-
This works for all model types (LLM, Whisper, NMT, Embeddings) and provides valuable insight into model performance and behavior.
|
|
243
|
-
|
|
244
166
|
### Completion
|
|
245
167
|
|
|
246
168
|
- `llama.cpp` with local files: [`examples/llamacpp-filesystem.ts`](examples/llamacpp-filesystem.ts)
|
|
247
169
|
- `llama.cpp` with Hyperdrive: [`examples/llamacpp-hyperdrive.ts`](examples/llamacpp-hyperdrive.ts)
|
|
248
170
|
- `llama.cpp` with HTTP: [`examples/llamacpp-http.ts`](examples/llamacpp-http.ts)
|
|
249
|
-
- `llama.cpp` with
|
|
250
|
-
- `llama.cpp` with tools/function calls: [`examples/llamacpp-tools.ts`](examples/llamacpp-tools.ts)
|
|
251
|
-
- `llama.cpp` with tools and routing: [`examples/llamacpp-tools-routing.ts`](examples/llamacpp-tools-routing.ts)
|
|
171
|
+
- `llama.cpp` with tools/function calls: [`examples/llamacpp-native-tools.ts`](examples/llamacpp-native-tools.ts)
|
|
252
172
|
- `llama.cpp` with multimodal inference: [`examples/llamacpp-multimodal.ts`](examples/llamacpp-multimodal.ts)
|
|
253
173
|
- `llama.cpp` with KV cache: [`examples/kv-cache-example.ts`](examples/kv-cache-example.ts)
|
|
254
174
|
|
|
255
175
|
### Transcription
|
|
256
176
|
|
|
257
|
-
- `whisper.cpp` transcription: [`examples/whispercpp-
|
|
258
|
-
- Microphone recording: [`examples/microphone-record
|
|
177
|
+
- `whisper.cpp` transcription: [`examples/whispercpp-filesystem.ts`](examples/whispercpp-filesystem.ts)
|
|
178
|
+
- Microphone recording: [`examples/whispercpp-microphone-record.ts`](examples/whispercpp-microphone-record.ts)
|
|
259
179
|
|
|
260
180
|
### Embeddings
|
|
261
181
|
|
|
262
|
-
-
|
|
263
|
-
- RAG
|
|
264
|
-
- RAG with workspaces: [`examples/rag-hyperdb-workspaces.ts`](examples/rag-hyperdb-workspaces.ts) _(Demonstrates workspace isolation and chunking)_
|
|
265
|
-
- RAG
|
|
266
|
-
- RAG
|
|
267
|
-
- RAG
|
|
182
|
+
- Single and batch embeddings: [`examples/embed-hyperdrive.ts`](examples/embed-hyperdrive.ts)
|
|
183
|
+
- RAG with HyperDB: [`examples/rag/rag-hyperdb.ts`](examples/rag/rag-hyperdb.ts)
|
|
184
|
+
- RAG with workspaces: [`examples/rag/rag-hyperdb-workspaces.ts`](examples/rag/rag-hyperdb-workspaces.ts) _(Demonstrates workspace isolation and chunking)_
|
|
185
|
+
- RAG with LanceDB (desktop only): [`examples/rag/rag-lancedb.ts`](examples/rag/rag-lancedb.ts)
|
|
186
|
+
- RAG with ChromaDB (desktop only): [`examples/rag/rag-chromadb.ts`](examples/rag/rag-chromadb.ts) _(Note: Requires ChromaDB server running)_
|
|
187
|
+
- RAG with SQLite-Vector (desktop only): [`examples/rag/rag-sqlite.ts`](examples/rag/rag-sqlite.ts) _(Note: Uses SQLite-Vector WASM)_
|
|
268
188
|
|
|
269
189
|
### Translation
|
|
270
190
|
|
|
271
|
-
- Marian OPUS translation: [`examples/translation-opus.ts`](examples/translation-opus.ts)
|
|
272
|
-
- Indic language translation: [`examples/translation-indic.ts`](examples/translation-indic.ts)
|
|
273
|
-
- LLM-based translation: [`examples/translation-llm.ts`](examples/translation-llm.ts)
|
|
191
|
+
- Marian OPUS translation: [`examples/translation/translation-opus.ts`](examples/translation/translation-opus.ts)
|
|
192
|
+
- Indic language translation: [`examples/translation/translation-indic.ts`](examples/translation/translation-indic.ts)
|
|
193
|
+
- LLM-based translation: [`examples/translation/translation-llm.ts`](examples/translation/translation-llm.ts)
|
|
274
194
|
|
|
275
|
-
### Text
|
|
195
|
+
### Text-to-Speech
|
|
276
196
|
|
|
277
|
-
-
|
|
278
|
-
|
|
279
|
-
**Note**: To run the text-to-speech example, espeak data is required which can be obtained from [here](https://github.com/espeak-ng/espeak-ng).
|
|
197
|
+
- TTS with [eSpeak](https://github.com/espeak-ng/espeak-ng): [`examples/text-to-speech.ts`](examples/text-to-speech.ts)
|
|
280
198
|
|
|
281
199
|
### Multimodel
|
|
282
200
|
|
|
@@ -284,22 +202,39 @@ This works for all model types (LLM, Whisper, NMT, Embeddings) and provides valu
|
|
|
284
202
|
|
|
285
203
|
### Delegated inference
|
|
286
204
|
|
|
287
|
-
- Provider
|
|
288
|
-
- Consumer
|
|
205
|
+
- Provider: [`examples/delegated-inference/provider.ts`](examples/delegated-inference/provider.ts)
|
|
206
|
+
- Consumer: [`examples/delegated-inference/consumer.ts`](examples/delegated-inference/consumer.ts)
|
|
289
207
|
|
|
290
|
-
|
|
291
|
-
|
|
208
|
+
> [!TIP]
|
|
209
|
+
> Set `QVAC_HYPERSWARM_SEED` env var to ensure that the provider uses the same keypair (i.e., public key doesn't change on every run).
|
|
292
210
|
|
|
293
|
-
|
|
211
|
+
> [!NOTE]
|
|
212
|
+
> Consumer does not handle reconnection yet.
|
|
294
213
|
|
|
295
|
-
|
|
214
|
+
### Logging
|
|
296
215
|
|
|
297
|
-
|
|
216
|
+
Stream real-time logs from the SDK server and native addons:
|
|
298
217
|
|
|
299
|
-
|
|
300
|
-
|
|
218
|
+
```ts
|
|
219
|
+
import { loggingStream, SDK_LOG_ID } from "@qvac/sdk";
|
|
220
|
+
|
|
221
|
+
// SDK server logs (general operations)
|
|
222
|
+
for await (const log of loggingStream({ id: SDK_LOG_ID })) {
|
|
223
|
+
console.log(`[${log.level}] ${log.namespace}: ${log.message}`);
|
|
224
|
+
}
|
|
301
225
|
|
|
302
|
-
|
|
226
|
+
// Addon logs per model (llamacpp, whispercpp, etc.)
|
|
227
|
+
for await (const log of loggingStream({ id: modelId })) {
|
|
228
|
+
console.log(`[${log.level}] ${log.namespace}: ${log.message}`);
|
|
229
|
+
}
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
- Log streaming: [`examples/logging-streaming.ts`](examples/logging-streaming.ts)
|
|
233
|
+
- Log with custom file transport: [`examples/logging-file-transport.ts`](examples/logging-file-transport.ts)
|
|
234
|
+
|
|
235
|
+
### Download Lifecycle
|
|
236
|
+
|
|
237
|
+
- Pause and resume download: [`examples/download-with-cancel.ts`](examples/download-with-cancel.ts)
|
|
303
238
|
|
|
304
239
|
### Blind Relays
|
|
305
240
|
|
|
@@ -308,17 +243,210 @@ Blind relays help establish peer connections through NAT/firewalls by routing tr
|
|
|
308
243
|
- Model downloads via Hyperdrive: [`examples/download-with-blind-relays.ts`](./examples/download-with-blind-relays.ts)
|
|
309
244
|
- Delegated inference: You can reuse the same pattern for delegated inference by calling `setConfig({ swarmRelays: [...relayPublicKeys] })` before starting your provider/consumer.
|
|
310
245
|
|
|
246
|
+
> [!NOTE]
|
|
247
|
+
> The examples use mock relay keys. For real deployments, you **must** use your own relay servers or trusted public relays.
|
|
248
|
+
|
|
249
|
+
### Sharded Models
|
|
250
|
+
|
|
251
|
+
- Load sharded models from Hyperdrive: [`examples/llamacpp-sharded.ts`](examples/llamacpp-sharded.ts)
|
|
252
|
+
- Load models from local filesystem: [`examples/llamacpp-filesystem.ts`](examples/llamacpp-filesystem.ts)
|
|
253
|
+
|
|
254
|
+
> [!NOTE]
|
|
255
|
+
> Sharded models automatically download all parts sequentially with detailed progress tracking per shard. For local sharded models, pass the path to the first shard file (e.g., `model-00001-of-00005.gguf`), and remaining shards will be loaded automatically.
|
|
256
|
+
|
|
257
|
+
## Basic flow
|
|
258
|
+
|
|
259
|
+
```mermaid
|
|
260
|
+
sequenceDiagram
|
|
261
|
+
participant User as User
|
|
262
|
+
participant SDK as QVAC SDK
|
|
263
|
+
participant RPC as RPC Client (singleton)
|
|
264
|
+
participant Worker as Bare Worker (singleton)
|
|
265
|
+
|
|
266
|
+
User->>SDK: Call loadModel("llama-3", options)
|
|
267
|
+
SDK->>RPC: Create runtime-specific RPC client
|
|
268
|
+
RPC->>Worker: Spawn worker (if needed) and send loadModel request
|
|
269
|
+
Worker->>Worker: Download and load model into memory
|
|
270
|
+
Worker->>RPC: Return success response
|
|
271
|
+
RPC->>SDK: Model loaded successfully
|
|
272
|
+
SDK->>User: loadModel() resolves with modelId
|
|
273
|
+
```
|
|
274
|
+
|
|
311
275
|
**Note**: The example uses mock relay keys. In real deployments, you **must** use your own relay servers or trusted public relays.
|
|
312
276
|
|
|
313
|
-
|
|
277
|
+
### Hot Config Reload
|
|
278
|
+
|
|
279
|
+
Hot config reload allows you to update model configurations on-the-fly without unloading the model. Pass `modelId` (instead of `modelSrc`) to `loadModel` with the `modelType` and new `modelConfig` to apply changes instantly.
|
|
280
|
+
|
|
281
|
+
- Config reload using whisper: [`examples/config-reload.ts`](examples/config-reload.ts)
|
|
282
|
+
|
|
283
|
+
**Note**: Config reload is currently supported for Whisper models. All config parameters except `contextParams` (GPU settings, flash attention) can be hot reloaded. `contextParams` are load-time only and require full model reload. More model types coming soon.
|
|
284
|
+
|
|
285
|
+
## Contributing
|
|
286
|
+
|
|
287
|
+
### Commit Message and PR Title Format
|
|
288
|
+
|
|
289
|
+
This repository enforces structured commit messages and PR titles to maintain consistency and generate changelogs automatically.
|
|
290
|
+
|
|
291
|
+
#### Format
|
|
292
|
+
|
|
293
|
+
**Commit messages:**
|
|
294
|
+
|
|
295
|
+
```
|
|
296
|
+
prefix[tags]?: subject
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
**PR titles:**
|
|
300
|
+
|
|
301
|
+
```
|
|
302
|
+
TICKET prefix[tags]: subject
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
#### Allowed Prefixes
|
|
306
|
+
|
|
307
|
+
- `feat` - New features or capabilities
|
|
308
|
+
- `fix` - Bug fixes
|
|
309
|
+
- `doc` - Documentation changes
|
|
310
|
+
- `test` - Test additions or modifications
|
|
311
|
+
- `mod` - Model-related changes
|
|
312
|
+
- `chore` - Maintenance tasks
|
|
313
|
+
- `infra` - CI/CD, tooling, infrastructure
|
|
314
|
+
|
|
315
|
+
#### Allowed Tags
|
|
316
|
+
|
|
317
|
+
Tags are optional:
|
|
318
|
+
|
|
319
|
+
- `[api]` - API changes (non-breaking)
|
|
320
|
+
- `[bc]` - Breaking changes (including breaking API changes)
|
|
321
|
+
|
|
322
|
+
#### Examples
|
|
323
|
+
|
|
324
|
+
**Valid commit messages:**
|
|
325
|
+
|
|
326
|
+
```bash
|
|
327
|
+
feat: add RAG support for LanceDB
|
|
328
|
+
fix[api]: fix completion stream error handling
|
|
329
|
+
doc: update installation instructions
|
|
330
|
+
feat[bc]: redesign loadModel signature
|
|
331
|
+
chore: update dependencies
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
**Valid PR titles:**
|
|
335
|
+
|
|
336
|
+
```bash
|
|
337
|
+
QVAC-123 feat: add RAG support for LanceDB
|
|
338
|
+
QVAC-456 fix[api]: fix completion stream error handling
|
|
339
|
+
QVAC-789 doc: update installation instructions
|
|
340
|
+
QVAC-101 feat[bc]: redesign loadModel signature
|
|
341
|
+
```
|
|
342
|
+
|
|
343
|
+
#### Code Examples Requirements
|
|
344
|
+
|
|
345
|
+
When creating PRs with specific tags, you must include code examples in the PR description:
|
|
346
|
+
|
|
347
|
+
**`[bc]` tag requirements:**
|
|
348
|
+
|
|
349
|
+
Must include BEFORE/AFTER code examples showing the migration path:
|
|
350
|
+
|
|
351
|
+
````markdown
|
|
352
|
+
## BC Changes
|
|
353
|
+
|
|
354
|
+
**BEFORE:**
|
|
355
|
+
|
|
356
|
+
```typescript
|
|
357
|
+
const model = await loadModel("model-path");
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
**AFTER:**
|
|
361
|
+
|
|
362
|
+
```typescript
|
|
363
|
+
const modelId = await loadModel("model-path", { modelType: "llm" });
|
|
364
|
+
```
|
|
365
|
+
````
|
|
366
|
+
|
|
367
|
+
Or using inline comments:
|
|
368
|
+
|
|
369
|
+
````markdown
|
|
370
|
+
```typescript
|
|
371
|
+
// old
|
|
372
|
+
const model = await loadModel("model-path");
|
|
373
|
+
|
|
374
|
+
// new
|
|
375
|
+
const modelId = await loadModel("model-path", { modelType: "llm" });
|
|
376
|
+
```
|
|
377
|
+
````
|
|
378
|
+
|
|
379
|
+
**`[api]` tag requirements (non-breaking):**
|
|
380
|
+
|
|
381
|
+
Must include at least one fenced code block showing the new API usage:
|
|
382
|
+
|
|
383
|
+
````markdown
|
|
384
|
+
## New API
|
|
385
|
+
|
|
386
|
+
```typescript
|
|
387
|
+
// New completion API with streaming support
|
|
388
|
+
for await (const token of completion({
|
|
389
|
+
modelId,
|
|
390
|
+
history: [{ role: "user", content: "Hello!" }],
|
|
391
|
+
}).tokenStream) {
|
|
392
|
+
process.stdout.write(token);
|
|
393
|
+
}
|
|
394
|
+
```
|
|
395
|
+
````
|
|
396
|
+
|
|
397
|
+
#### Validation
|
|
398
|
+
|
|
399
|
+
- **Commit messages** are validated automatically via Husky commit-msg hook
|
|
400
|
+
- **PR titles and descriptions** are validated via GitHub Actions on PR creation/update
|
|
401
|
+
- Invalid commits or PRs will be rejected with helpful error messages
|
|
402
|
+
- **Auto-skipped commits:** The following Git-generated commits bypass validation:
|
|
403
|
+
- Merge commits (e.g., `Merge pull request #123`)
|
|
404
|
+
- Version bumps (e.g., `1.0.0`, `v1.0.0`)
|
|
405
|
+
- Revert commits (e.g., `Revert "feat: add feature"`)
|
|
406
|
+
- Squash commits (e.g., `squash! fix: bug fix`)
|
|
407
|
+
|
|
408
|
+
#### Generating Changelogs
|
|
409
|
+
|
|
410
|
+
Once your PRs are merged into `dev`, you can generate a changelog:
|
|
411
|
+
|
|
412
|
+
```bash
|
|
413
|
+
npm run changelog:generate
|
|
414
|
+
```
|
|
415
|
+
|
|
416
|
+
This will:
|
|
417
|
+
|
|
418
|
+
1. Compare versions between `dev` and `main` branches
|
|
419
|
+
2. Collect all merged PRs
|
|
420
|
+
3. Parse and classify each PR by prefix
|
|
421
|
+
4. Generate `changelog/<version>/CHANGELOG.md`
|
|
422
|
+
5. Generate `changelog/<version>/breaking.md` for BC changes (with code examples)
|
|
423
|
+
6. Generate `changelog/<version>/api.md` for API changes (with code examples)
|
|
424
|
+
|
|
425
|
+
**Note:** Requires a GitHub token (`GITHUB_TOKEN` or `GH_TOKEN` environment variable) to fetch PR metadata.
|
|
426
|
+
|
|
427
|
+
## Build
|
|
428
|
+
|
|
429
|
+
Use the [Bun](https://bun.sh/) package manager:
|
|
430
|
+
|
|
431
|
+
```bash
|
|
432
|
+
bun i
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
```bash
|
|
436
|
+
bun run build # or `watch` for hotreload
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
```bash
|
|
440
|
+
bun run build:pack
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
This outputs a tarball under `dist/qvac-sdk-{version}.tgz` that you can install in your project, e.g.:
|
|
444
|
+
|
|
445
|
+
```bash
|
|
446
|
+
npm i path/to/qvac-sdk-0.3.0.tgz
|
|
447
|
+
```
|
|
314
448
|
|
|
315
|
-
|
|
316
|
-
- Add a `process.exit()` call at the end, we're working on it.
|
|
449
|
+
## More Resources
|
|
317
450
|
|
|
318
|
-
-
|
|
319
|
-
|
|
320
|
-
- Open the `ios` folder in Xcode
|
|
321
|
-
- Start the app with the play ▶️ button
|
|
322
|
-
- View -> Debug area -> Activate console
|
|
323
|
-
- You should see the system logs on the right pane
|
|
324
|
-
- ⚠️ This only works when starting the app from Xcode
|
|
451
|
+
- [Comprehensive documentation of this SDK](https://qvac.tether.dev/docs/sdk)
|
|
452
|
+
- [Package at NPM](https://www.npmjs.com/package/@qvac/sdk)
|