@qvac/sdk 0.4.1 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (179) hide show
  1. package/README.md +328 -200
  2. package/dist/client/api/completion-stream.d.ts +15 -5
  3. package/dist/client/api/completion-stream.js +50 -14
  4. package/dist/client/api/embed.d.ts +17 -3
  5. package/dist/client/api/embed.js +0 -8
  6. package/dist/client/api/get-model-info.d.ts +1 -1
  7. package/dist/client/api/index.d.ts +0 -1
  8. package/dist/client/api/index.js +0 -1
  9. package/dist/client/api/load-model.d.ts +37 -11
  10. package/dist/client/api/load-model.js +29 -117
  11. package/dist/client/api/logging-stream.d.ts +6 -3
  12. package/dist/client/api/logging-stream.js +6 -3
  13. package/dist/client/api/unload-model.js +5 -2
  14. package/dist/client/config-loader/config-utils.d.ts +5 -0
  15. package/dist/client/config-loader/config-utils.js +20 -0
  16. package/dist/client/config-loader/resolve-config.bare.d.ts +9 -0
  17. package/dist/client/config-loader/resolve-config.bare.js +87 -0
  18. package/dist/client/config-loader/resolve-config.expo.d.ts +11 -0
  19. package/dist/client/config-loader/resolve-config.expo.js +62 -0
  20. package/dist/client/config-loader/resolve-config.node.d.ts +9 -0
  21. package/dist/client/config-loader/resolve-config.node.js +118 -0
  22. package/dist/client/init-hooks.d.ts +17 -3
  23. package/dist/client/init-hooks.js +37 -18
  24. package/dist/client/logging-stream-registry.d.ts +1 -1
  25. package/dist/client/logging-stream-registry.js +14 -13
  26. package/dist/client/rpc/bare-client.d.ts +10 -14
  27. package/dist/client/rpc/bare-client.js +28 -2
  28. package/dist/client/rpc/bare-rpc-client.d.ts +1 -1
  29. package/dist/client/rpc/bare-rpc-client.js +16 -13
  30. package/dist/client/rpc/expo-rpc-client.d.ts +2 -2
  31. package/dist/client/rpc/expo-rpc-client.js +15 -12
  32. package/dist/client/rpc/node-rpc-client.d.ts +1 -1
  33. package/dist/client/rpc/node-rpc-client.js +10 -7
  34. package/dist/client/rpc/rpc-client.js +16 -15
  35. package/dist/examples/config/config-path.d.ts +8 -0
  36. package/dist/examples/config/config-path.js +13 -0
  37. package/dist/examples/config-reload.d.ts +2 -0
  38. package/dist/examples/config-reload.js +49 -0
  39. package/dist/examples/download-with-blind-relays.js +20 -20
  40. package/dist/examples/embed-hyperdrive.js +35 -3
  41. package/dist/examples/kv-cache-cleanup.js +1 -1
  42. package/dist/examples/kv-cache-custom-key.js +1 -1
  43. package/dist/examples/kv-cache-example.js +1 -1
  44. package/dist/examples/llamacpp-hyperdrive.js +1 -1
  45. package/dist/examples/llamacpp-sharded.js +1 -1
  46. package/dist/examples/logging-file-transport.d.ts +2 -0
  47. package/dist/examples/{llamacpp-file-logging.js → logging-file-transport.js} +3 -7
  48. package/dist/examples/logging-streaming.d.ts +2 -0
  49. package/dist/examples/logging-streaming.js +75 -0
  50. package/dist/examples/mcp-websearch.d.ts +13 -0
  51. package/dist/examples/mcp-websearch.js +176 -0
  52. package/dist/index.d.ts +4 -4
  53. package/dist/index.js +2 -2
  54. package/dist/logging/addon.d.ts +6 -0
  55. package/dist/logging/addon.js +59 -0
  56. package/dist/logging/base-logger.js +24 -14
  57. package/dist/logging/client-logger.d.ts +3 -0
  58. package/dist/logging/client-logger.js +13 -0
  59. package/dist/logging/index.d.ts +6 -1
  60. package/dist/logging/index.js +6 -1
  61. package/dist/logging/logger.d.ts +0 -2
  62. package/dist/logging/logger.js +0 -3
  63. package/dist/logging/namespaces.d.ts +11 -0
  64. package/dist/logging/namespaces.js +11 -0
  65. package/dist/logging/registry.d.ts +26 -0
  66. package/dist/logging/registry.js +36 -0
  67. package/dist/logging/server-logger.d.ts +3 -0
  68. package/dist/logging/server-logger.js +16 -0
  69. package/dist/logging/stream-logger.d.ts +1 -1
  70. package/dist/logging/stream-logger.js +6 -3
  71. package/dist/logging/types.d.ts +2 -0
  72. package/dist/models/hyperdrive/models.d.ts +789 -51
  73. package/dist/models/hyperdrive/models.js +979 -200
  74. package/dist/schemas/common.d.ts +86 -23
  75. package/dist/schemas/common.js +0 -3
  76. package/dist/schemas/download-asset.d.ts +5 -5
  77. package/dist/schemas/embed.d.ts +3 -3
  78. package/dist/schemas/embed.js +9 -2
  79. package/dist/schemas/get-model-info.d.ts +2 -2
  80. package/dist/schemas/index.d.ts +5 -1
  81. package/dist/schemas/index.js +5 -1
  82. package/dist/schemas/json-schema.d.ts +16 -0
  83. package/dist/schemas/json-schema.js +2 -0
  84. package/dist/schemas/llamacpp-config.d.ts +5 -4
  85. package/dist/schemas/llamacpp-config.js +7 -5
  86. package/dist/schemas/load-model.d.ts +931 -211
  87. package/dist/schemas/load-model.js +7 -1
  88. package/dist/schemas/logging-stream.d.ts +10 -3
  89. package/dist/schemas/logging-stream.js +4 -3
  90. package/dist/schemas/mcp-adapter.d.ts +47 -0
  91. package/dist/schemas/mcp-adapter.js +1 -0
  92. package/dist/schemas/model-src-utils.d.ts +8 -6
  93. package/dist/schemas/model-src-utils.js +8 -3
  94. package/dist/schemas/reload-config.d.ts +344 -0
  95. package/dist/schemas/reload-config.js +42 -0
  96. package/dist/schemas/sdk-config.d.ts +22 -0
  97. package/dist/schemas/sdk-config.js +33 -0
  98. package/dist/schemas/sdk-errors-client.d.ts +4 -0
  99. package/dist/schemas/sdk-errors-client.js +20 -0
  100. package/dist/schemas/sdk-errors-server.d.ts +3 -0
  101. package/dist/schemas/sdk-errors-server.js +15 -0
  102. package/dist/schemas/tools.d.ts +3 -0
  103. package/dist/schemas/translation-config.d.ts +81 -0
  104. package/dist/schemas/translation-config.js +3 -4
  105. package/dist/schemas/whispercpp-config.d.ts +9 -1
  106. package/dist/schemas/whispercpp-config.js +2 -1
  107. package/dist/server/bare/addons/llamacpp/create-model.js +13 -4
  108. package/dist/server/bare/addons/llamacpp/handlers/embed.d.ts +9 -1
  109. package/dist/server/bare/addons/llamacpp/handlers/embed.js +18 -4
  110. package/dist/server/bare/addons/llamacpp/kv-cache-utils.js +6 -4
  111. package/dist/server/bare/addons/load-model.js +10 -4
  112. package/dist/server/bare/addons/unload-model.js +6 -3
  113. package/dist/server/bare/addons/whispercpp/handlers/transcribe-stream.js +56 -37
  114. package/dist/server/bare/delegate-rpc-client.js +12 -10
  115. package/dist/server/bare/hyperswarm.js +7 -5
  116. package/dist/server/bare/rag-hyperdb/rag-manager.js +3 -1
  117. package/dist/server/bare/registry/config-registry.d.ts +13 -4
  118. package/dist/server/bare/registry/config-registry.js +42 -10
  119. package/dist/server/bare/registry/logging-stream-registry.d.ts +24 -7
  120. package/dist/server/bare/registry/logging-stream-registry.js +99 -20
  121. package/dist/server/bare/registry/model-registry.d.ts +3 -0
  122. package/dist/server/bare/registry/model-registry.js +9 -6
  123. package/dist/server/rpc/create-server.js +5 -3
  124. package/dist/server/rpc/handle-request.js +30 -16
  125. package/dist/server/rpc/handlers/cancelHandler.js +3 -1
  126. package/dist/server/rpc/handlers/completion-stream-delegated.js +4 -2
  127. package/dist/server/rpc/handlers/delete-cache.js +3 -1
  128. package/dist/server/rpc/handlers/download-asset.js +3 -1
  129. package/dist/server/rpc/handlers/embed.js +3 -1
  130. package/dist/server/rpc/handlers/index.d.ts +0 -1
  131. package/dist/server/rpc/handlers/index.js +0 -2
  132. package/dist/server/rpc/handlers/load-model/download-manager.js +4 -2
  133. package/dist/server/rpc/handlers/load-model/http.js +20 -18
  134. package/dist/server/rpc/handlers/load-model/hyperdrive.js +21 -19
  135. package/dist/server/rpc/handlers/load-model/resolve.js +7 -5
  136. package/dist/server/rpc/handlers/load-model-delegated.d.ts +2 -2
  137. package/dist/server/rpc/handlers/load-model-delegated.js +10 -8
  138. package/dist/server/rpc/handlers/load-model.js +55 -3
  139. package/dist/server/rpc/handlers/logging-stream.js +4 -4
  140. package/dist/server/rpc/handlers/provideHandler/connection.js +12 -10
  141. package/dist/server/rpc/handlers/provideHandler/handler.js +18 -16
  142. package/dist/server/rpc/handlers/provideHandler/proxy.js +8 -6
  143. package/dist/server/rpc/handlers/rag.js +3 -1
  144. package/dist/server/rpc/handlers/stopProvideHandler.js +3 -1
  145. package/dist/server/rpc/handlers/transcribe-stream.js +3 -1
  146. package/dist/server/rpc/handlers/translate.js +3 -1
  147. package/dist/server/rpc/handlers/unload-model.js +4 -2
  148. package/dist/server/utils/audio/decoder.d.ts +2 -1
  149. package/dist/server/utils/audio/decoder.js +48 -11
  150. package/dist/server/utils/cache.js +7 -5
  151. package/dist/server/utils/config-transform.d.ts +3 -0
  152. package/dist/server/utils/config-transform.js +21 -0
  153. package/dist/server/utils/index.d.ts +1 -0
  154. package/dist/server/utils/index.js +1 -0
  155. package/dist/server/worker.js +41 -16
  156. package/dist/utils/errors-client.d.ts +12 -0
  157. package/dist/utils/errors-client.js +20 -0
  158. package/dist/utils/errors-server.d.ts +9 -0
  159. package/dist/utils/errors-server.js +16 -0
  160. package/dist/utils/getProcessSafe.d.ts +7 -0
  161. package/dist/utils/getProcessSafe.js +17 -0
  162. package/dist/utils/mcp-adapter.d.ts +12 -0
  163. package/dist/utils/mcp-adapter.js +103 -0
  164. package/dist/utils/object.d.ts +2 -0
  165. package/dist/utils/object.js +3 -0
  166. package/dist/utils/runtime.d.ts +30 -0
  167. package/dist/utils/runtime.js +48 -0
  168. package/dist/utils/tool-helpers.d.ts +10 -2
  169. package/dist/utils/tool-helpers.js +24 -11
  170. package/package.json +46 -33
  171. package/dist/client/api/set-config.d.ts +0 -7
  172. package/dist/client/api/set-config.js +0 -20
  173. package/dist/client/config-manager.d.ts +0 -4
  174. package/dist/client/config-manager.js +0 -8
  175. package/dist/examples/llamacpp-file-logging.d.ts +0 -2
  176. package/dist/schemas/set-config.d.ts +0 -21
  177. package/dist/schemas/set-config.js +0 -14
  178. package/dist/server/rpc/handlers/set-config.d.ts +0 -3
  179. package/dist/server/rpc/handlers/set-config.js +0 -18
package/README.md CHANGED
@@ -1,14 +1,25 @@
1
- [![QVAC logo](https://tether-2.gitbook.io/qvac-by-tether-or-reference/y2AsIh94H9DJ1pIBCNj9/~gitbook/image?url=https%3A%2F%2F4184520186-files.gitbook.io%2F~%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252Fnu0sKQdYg5T0N88iY6ZN%252Fsites%252Fsite_Fu2B8%252Flogo%252Fto5sbpix5Uz6MFPzPwY1%252FVector.png%3Falt%3Dmedia%26token%3D27902e06-6941-413c-a199-1e2780444d1a&width=260&dpr=2&quality=100&sign=8040ed76&sv=2)](https://tether-2.gitbook.io/qvac-by-tether-or-reference/y2AsIh94H9DJ1pIBCNj9)
1
+ [![QVAC logo](docs/logo.avif)](https://qvac.tether.dev)
2
2
 
3
3
  ---
4
4
 
5
- **QVAC** is an open-source, cross-platform system for building local-first, peer-to-peer **AI applications**. QVAC runs on [**Bare**](https://bare.pears.com) by [Holepunch](https://holepunch.to), a lightweight, cross-platform JavaScript runtime.
5
+ **QVAC** is an open-source, cross-platform **ecosystem** for local-first, peer-to-peer **AI**. QVAC runs on [**Bare**](https://bare.pears.com) by [Holepunch](https://holepunch.to), a lightweight, cross-platform JavaScript runtime.
6
6
 
7
7
  # QVAC SDK
8
8
 
9
- **QVAC SDK** is the canonical entry point to QVAC. Written in TypeScript, it provides all QVAC capabilities through a unified interface while also abstracting away the complexity of running your application in a JS environment other than Bare.
9
+ **QVAC SDK** is the canonical entry point to develop AI applications with QVAC.
10
10
 
11
- For the comprehensive reference, [see official QVAC documentation](https://tether-2.gitbook.io/qvac-by-tether-or-reference/y2AsIh94H9DJ1pIBCNj9).
11
+ > _Part of **QVAC** ecosystem_
12
+ > <br>
13
+ > <sup>
14
+ > <a href="https://qvac.tether.dev" >Home</a> &nbsp;•&nbsp;
15
+ > <a href="https://qvac.tether.dev/docs" >Docs</a> &nbsp;•&nbsp;
16
+ > <a href="https://qvacbytether.featurebase.app" >Support</a> &nbsp;•&nbsp;
17
+ > <a href="https://discord.com/invite/tetherdev" >Discord</a>
18
+ > </sup>
19
+
20
+ Written in TypeScript, it provides all QVAC capabilities through a unified interface while also abstracting away the complexity of running your application in a JS environment other than Bare.
21
+
22
+ For the comprehensive reference, [see official QVAC documentation](https://qvac.tether.dev/docs).
12
23
 
13
24
  ## Requirements
14
25
 
@@ -16,65 +27,29 @@ Supported JS environments: Bare, Node.js, Expo and Bun.
16
27
 
17
28
  ## Installation
18
29
 
19
- ### Steps
20
-
21
- 1. [Authenticate to Tether’s GitHub packages](#step-1-authenticate-to-tethers-github-packages)
22
- 2. [Install package and dependencies](#step-2-install-package-and-dependencies)
23
-
24
- ### Step 1: authenticate to Tether’s GitHub packages
25
-
26
- 1. [Create a GitHub personal access token (classic) with scope `read:packages`](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic).
27
-
28
- 2. Add the token to your environment variables. For example, on macOS/Linux:
29
-
30
30
  ```bash
31
- export NPM_TOKEN=<token_generated_in_previous_step>
32
- ```
33
-
34
- 3. In the root of your project, create file `.npmrc` with the following content:
35
-
36
- ```ini
37
- @qvac:registry=https://registry.npmjs.org
38
- //registry.npmjs.org/:_authToken=${NPM_TOKEN}
31
+ npm install @qvac/sdk
39
32
  ```
40
33
 
41
- _Alternatively, you can create the `.npmrc` file in your home directory, for user-level (global) configuration._
34
+ ### Linux
42
35
 
43
- 4. Confirm authentication to Tether’s GitHub packages:
36
+ OS peer dependency:
44
37
 
45
38
  ```bash
46
- npm whoami --registry=https://npm.pkg.github.com
39
+ apt install vulkan-sdk
47
40
  ```
48
41
 
49
- It should return your GitHub username.
50
-
51
- ### Step 2: Install package and dependencies
42
+ ### Expo
52
43
 
53
- 1. Install in your project using the package manager of your choice:
54
-
55
- ```bash
56
- npm i @qvac/sdk
57
- ```
58
-
59
- > [!TIP]
60
- > If you can't get the package via Tether's private GitHub package registry, see section [Build from source](#build-from-source).
61
-
62
- #### On Linux
63
-
64
- > [!IMPORTANT]
65
- > If you're on Linux ensure that the Vulkan SDK is installed (e.g. `apt install vulkan-sdk` on Ubuntu)
66
-
67
- #### On Expo
68
-
69
- 1. Install the peer dependencies for Expo/React Native support:
44
+ 1. Peer dependencies:
70
45
 
71
46
  ```bash
72
47
  npm i expo-file-system react-native-bare-kit
73
48
  ```
74
49
 
75
- 2. If you are on Android, bump your `minSdkVersion` to 29. You can do so by adding `ext { minSdkVersion=29 }` to your `android/build.gradle` or by using `expo-build-properties`.
50
+ 2. On Android, bump `minSdkVersion` to 29, by adding `ext { minSdkVersion=29 }` to `android/build.gradle` or using `expo-build-properties`.
76
51
 
77
- 3. Add the QVAC Expo plugin to your `app.json`:
52
+ 3. Add the QVAC Expo plugin to `app.json`:
78
53
 
79
54
  ```js
80
55
  export default {
@@ -101,78 +76,80 @@ npx expo run:android --device
101
76
  > [!IMPORTANT]
102
77
  > Due to limitations with `llamacpp`, QVAC currently does not run on emulators. You **must** use a physical device.
103
78
 
104
- ## Quickstart
105
-
106
- You can run the self-contained example below:
79
+ ## Usage
107
80
 
108
81
  ```js
109
- import { loadModel, completion, unloadModel } from "@qvac/sdk";
110
-
111
- // Load model into memory
112
- const modelId = await loadModel(
113
- "pear://afa79ee07c0a138bb9f11bfaee771fb1bdfca8c82d961cff0474e49827bd1de3/Llama-3.2-1B-Instruct-Q4_0.gguf",
114
- {
82
+ import {
83
+ completion,
84
+ LLAMA_3_2_1B_INST_Q4_0,
85
+ loadModel,
86
+ downloadAsset,
87
+ unloadModel,
88
+ VERBOSITY,
89
+ } from "@qvac/sdk";
90
+ try {
91
+ // First just cache the model
92
+ await downloadAsset({
93
+ assetSrc: LLAMA_3_2_1B_INST_Q4_0,
94
+ onProgress: (progress) => {
95
+ console.log(progress);
96
+ },
97
+ });
98
+ // Then load it in memory from cache
99
+ const modelId = await loadModel({
100
+ modelSrc: LLAMA_3_2_1B_INST_Q4_0,
115
101
  modelType: "llm",
116
- },
117
- );
118
-
119
- // Use the loaded model
120
- const response = completion({
121
- modelId,
122
- history: [{ role: "user", content: "What is the capital of France?" }],
123
- stream: false,
124
- });
125
- const text = await response.text;
126
- console.log(text);
127
-
128
- // Enable KV cache for faster subsequent completions
129
- const cachedResponse = completion({
130
- modelId,
131
- history: [
132
- { role: "user", content: "What is the capital of France?" },
133
- { role: "assistant", content: "The capital of France is Paris." },
134
- { role: "user", content: "What about Germany?" },
135
- ],
136
- stream: true,
137
- kvCache: true, // enables caching for faster inference
138
- });
139
-
140
- // You can use the loaded model multiple times
141
-
142
- // Unload model to free up system resources
143
- await unloadModel({ modelId });
144
- process.exit();
102
+ modelConfig: {
103
+ device: "gpu",
104
+ ctx_size: 2048,
105
+ verbosity: VERBOSITY.ERROR,
106
+ },
107
+ });
108
+ const history = [
109
+ {
110
+ role: "user",
111
+ content: "Explain quantum computing in one sentence, use lots of emojis",
112
+ },
113
+ ];
114
+ const result = completion({ modelId, history, stream: true });
115
+ for await (const token of result.tokenStream) {
116
+ process.stdout.write(token);
117
+ }
118
+ const stats = await result.stats;
119
+ console.log("\n📊 Performance Stats:", stats);
120
+ // Change `clearStorage: true` to delete cached model files
121
+ await unloadModel({ modelId, clearStorage: false });
122
+ } catch (error) {
123
+ console.error("❌ Error:", error);
124
+ process.exit(1);
125
+ }
145
126
  ```
146
127
 
147
- **For more on how to use QVAC SDK, see [SDK at QVAC documentation](https://tether-2.gitbook.io/qvac-by-tether-or-reference/y2AsIh94H9DJ1pIBCNj9/sdk)**.
128
+ ## Functionalities
148
129
 
149
- ## Build from source
130
+ ### AI tasks
150
131
 
151
- Use the [Bun](https://bun.sh/) package manager:
132
+ - Completion: LLM inference via [`llama.cpp`](https://github.com/ggml-org/llama.cpp).
133
+ - Transcription: speech-to-text (ASR) via [`whisper.cpp`](https://github.com/ggml-org/whisper.cpp).
134
+ - Text embeddings: via `llama.cpp`, for RAG.
135
+ - Translation: between different languages.
136
+ - Text-to-Speech: TTS via ONNX.
137
+ - Multimodal: via [`llama.cpp`] — i.e., process and understand multiple types of media within the same conversation context.
138
+ - RAG: out-of-the-box retrieval-augmented generation workflow.
139
+ - Delegated inference: perform peer-to-peer edge inference via Holepunch stack.
152
140
 
153
- ```bash
154
- bun i
155
- ```
141
+ ### Utilities
156
142
 
157
- ```bash
158
- bun run build # or `watch` for hotreload
159
- ```
160
-
161
- ```bash
162
- bun run build:pack
163
- ```
164
-
165
- This outputs a tarball under `dist/qvac-sdk-{version}.tgz` that you can install in your project, e.g.:
166
-
167
- ```bash
168
- npm i path/to/qvac-sdk-0.3.0.tgz
169
- ```
143
+ - Logging: visibility into what's happening inside your models during loading, inference, and other operations.
144
+ - Download Lifecycle: pause and resume model downloads.
145
+ - Blind Relays: establish peer connections through NAT/firewalls by routing traffic through relay nodes.
146
+ - Sharded models: download a model that is sharded into multiple parts.
170
147
 
171
148
  ## Examples
172
149
 
173
- In the `./examples` subdirectory, you will find scripts demonstrating SDK usage. To try any of them:
150
+ In the `./examples` subdirectory, you will find scripts demonstrating how to use all SDK functionalities. To try any of them:
174
151
 
175
- 1. Follow the steps in section [Build from source](#build-from-source).
152
+ 1. Build the SDK from source (see [Build](#build) section).
176
153
  2. Run using Bare, Node.js, or Bun as the runtime:
177
154
 
178
155
  ```bash
@@ -186,97 +163,38 @@ node dist/examples/path/to/example.js
186
163
  bun run examples/path/to/example.ts
187
164
  ```
188
165
 
189
- ## Logging
190
-
191
- QVAC SDK includes built-in log propagation that streams model logs from the worker process to your client application in real-time. This gives you visibility into what's happening inside your models during loading, inference, and other operations.
192
-
193
- ### Usage
194
-
195
- Simply pass a logger when loading your model:
196
-
197
- ```js
198
- import { getLogger, loadModel, completion, unloadModel } from "@qvac/sdk";
199
-
200
- // Create a logger for your application
201
- const logger = getLogger("my-app", {
202
- level: "debug", // Set log level
203
- transports: [
204
- // Optional: add custom log handlers
205
- (level, namespace, message) => {
206
- console.log(`[${level.toUpperCase()}] ${namespace}: ${message}`);
207
- },
208
- ],
209
- });
210
-
211
- // Load model with logging - RPC stream starts automatically
212
- const modelId = await loadModel(
213
- "pear://afa79ee07c0a138bb9f11bfaee771fb1bdfca8c82d961cff0474e49827bd1de3/Llama-3.2-1B-Instruct-Q4_0.gguf",
214
- {
215
- modelType: "llm",
216
- logger, // Enable log streaming
217
- },
218
- );
219
-
220
- // Now you'll see model logs in your console as operations happen
221
- for await (const token of completion({
222
- modelId,
223
- history: [{ role: "user", content: "Hello!" }],
224
- }).tokenStream) {
225
- process.stdout.write(token);
226
- }
227
-
228
- await unloadModel({ modelId }); // Logging stream closes automatically
229
- ```
230
-
231
- ### What You'll See
232
-
233
- When logging is enabled, you'll see real-time logs from the underlying model libraries:
234
-
235
- ```
236
- [DEBUG] llamacpp:llm: Loading model weights...
237
- [INFO] llamacpp:llm: Model loaded successfully, vocab_size=32000
238
- [DEBUG] llamacpp:llm: Starting inference...
239
- [DEBUG] llamacpp:llm: Inference completed, tokens=12
240
- ```
241
-
242
- This works for all model types (LLM, Whisper, NMT, Embeddings) and provides valuable insight into model performance and behavior.
243
-
244
166
  ### Completion
245
167
 
246
168
  - `llama.cpp` with local files: [`examples/llamacpp-filesystem.ts`](examples/llamacpp-filesystem.ts)
247
169
  - `llama.cpp` with Hyperdrive: [`examples/llamacpp-hyperdrive.ts`](examples/llamacpp-hyperdrive.ts)
248
170
  - `llama.cpp` with HTTP: [`examples/llamacpp-http.ts`](examples/llamacpp-http.ts)
249
- - `llama.cpp` with file logging: [`examples/llamacpp-file-logging.ts`](examples/llamacpp-file-logging.ts)
250
- - `llama.cpp` with tools/function calls: [`examples/llamacpp-tools.ts`](examples/llamacpp-tools.ts)
251
- - `llama.cpp` with tools and routing: [`examples/llamacpp-tools-routing.ts`](examples/llamacpp-tools-routing.ts)
171
+ - `llama.cpp` with tools/function calls: [`examples/llamacpp-native-tools.ts`](examples/llamacpp-native-tools.ts)
252
172
  - `llama.cpp` with multimodal inference: [`examples/llamacpp-multimodal.ts`](examples/llamacpp-multimodal.ts)
253
173
  - `llama.cpp` with KV cache: [`examples/kv-cache-example.ts`](examples/kv-cache-example.ts)
254
174
 
255
175
  ### Transcription
256
176
 
257
- - `whisper.cpp` transcription: [`examples/whispercpp-transcription.ts`](examples/whispercpp-transcription.ts)
258
- - Microphone recording: [`examples/microphone-record-transcription.ts`](examples/microphone-record-transcription.ts)
177
+ - `whisper.cpp` transcription: [`examples/whispercpp-filesystem.ts`](examples/whispercpp-filesystem.ts)
178
+ - Microphone recording: [`examples/whispercpp-microphone-record.ts`](examples/whispercpp-microphone-record.ts)
259
179
 
260
180
  ### Embeddings
261
181
 
262
- - Embedding with Hyperdrive: [`examples/embed-hyperdrive.ts`](examples/embed-hyperdrive.ts)
263
- - RAG example with HyperDB: [`examples/rag-hyperdb.ts`](examples/rag-hyperdb.ts)
264
- - RAG with workspaces: [`examples/rag-hyperdb-workspaces.ts`](examples/rag-hyperdb-workspaces.ts) _(Demonstrates workspace isolation and chunking)_
265
- - RAG example with LanceDB (desktop only): [`examples/rag-lancedb.ts`](examples/rag-lancedb.ts)
266
- - RAG example with ChromaDB (desktop only): [`examples/rag-chromadb.ts`](examples/rag-chromadb.ts) _(Note: Requires ChromaDB server running)_
267
- - RAG example with SQLite-Vector (desktop only): [`examples/rag-sqlite.ts`](examples/rag-sqlite.ts) _(Note: Uses SQLite-Vector WASM)_
182
+ - Single and batch embeddings: [`examples/embed-hyperdrive.ts`](examples/embed-hyperdrive.ts)
183
+ - RAG with HyperDB: [`examples/rag/rag-hyperdb.ts`](examples/rag/rag-hyperdb.ts)
184
+ - RAG with workspaces: [`examples/rag/rag-hyperdb-workspaces.ts`](examples/rag/rag-hyperdb-workspaces.ts) _(Demonstrates workspace isolation and chunking)_
185
+ - RAG with LanceDB (desktop only): [`examples/rag/rag-lancedb.ts`](examples/rag/rag-lancedb.ts)
186
+ - RAG with ChromaDB (desktop only): [`examples/rag/rag-chromadb.ts`](examples/rag/rag-chromadb.ts) _(Note: Requires ChromaDB server running)_
187
+ - RAG with SQLite-Vector (desktop only): [`examples/rag/rag-sqlite.ts`](examples/rag/rag-sqlite.ts) _(Note: Uses SQLite-Vector WASM)_
268
188
 
269
189
  ### Translation
270
190
 
271
- - Marian OPUS translation: [`examples/translation-opus.ts`](examples/translation-opus.ts)
272
- - Indic language translation: [`examples/translation-indic.ts`](examples/translation-indic.ts)
273
- - LLM-based translation: [`examples/translation-llm.ts`](examples/translation-llm.ts)
191
+ - Marian OPUS translation: [`examples/translation/translation-opus.ts`](examples/translation/translation-opus.ts)
192
+ - Indic language translation: [`examples/translation/translation-indic.ts`](examples/translation/translation-indic.ts)
193
+ - LLM-based translation: [`examples/translation/translation-llm.ts`](examples/translation/translation-llm.ts)
274
194
 
275
- ### Text to Speech
195
+ ### Text-to-Speech
276
196
 
277
- - Text to Speech: [`examples/text-to-speech.ts`](examples/text-to-speech.ts)
278
-
279
- **Note**: To run the text-to-speech example, espeak data is required which can be obtained from [here](https://github.com/espeak-ng/espeak-ng).
197
+ - TTS with [eSpeak](https://github.com/espeak-ng/espeak-ng): [`examples/text-to-speech.ts`](examples/text-to-speech.ts)
280
198
 
281
199
  ### Multimodel
282
200
 
@@ -284,22 +202,39 @@ This works for all model types (LLM, Whisper, NMT, Embeddings) and provides valu
284
202
 
285
203
  ### Delegated inference
286
204
 
287
- - Provider example: [`examples/delegated-inference/provider.ts`](examples/delegated-inference/provider.ts)
288
- - Consumer example: [`examples/delegated-inference/consumer.ts`](examples/delegated-inference/consumer.ts)
205
+ - Provider: [`examples/delegated-inference/provider.ts`](examples/delegated-inference/provider.ts)
206
+ - Consumer: [`examples/delegated-inference/consumer.ts`](examples/delegated-inference/consumer.ts)
289
207
 
290
- **Note**: Set the `QVAC_HYPERSWARM_SEED` env var to ensure that the provider's uses the same keypair (i.e. the public key doesn't change on every run).
291
- **Note**: ⚠️ Consumer does not handle reconnection yet.
208
+ > [!TIP]
209
+ > Set `QVAC_HYPERSWARM_SEED` env var to ensure that the provider uses the same keypair (i.e., public key doesn't change on every run).
292
210
 
293
- ### Download Lifecycle
211
+ > [!NOTE]
212
+ > Consumer does not handle reconnection yet.
294
213
 
295
- - Pause and resume download: [`examples/download-with-cancel.ts`](examples/download-with-cancel.ts)
214
+ ### Logging
296
215
 
297
- ### Sharded Models
216
+ Stream real-time logs from the SDK server and native addons:
298
217
 
299
- - Load sharded models from Hyperdrive: [`examples/llamacpp-sharded.ts`](examples/llamacpp-sharded.ts)
300
- - Load models from local filesystem: [`examples/llamacpp-filesystem.ts`](examples/llamacpp-filesystem.ts)
218
+ ```ts
219
+ import { loggingStream, SDK_LOG_ID } from "@qvac/sdk";
220
+
221
+ // SDK server logs (general operations)
222
+ for await (const log of loggingStream({ id: SDK_LOG_ID })) {
223
+ console.log(`[${log.level}] ${log.namespace}: ${log.message}`);
224
+ }
301
225
 
302
- **Note**: Sharded models automatically download all parts sequentially with detailed progress tracking per shard. For local sharded models, pass the path to the first shard file (e.g., `model-00001-of-00005.gguf`), and remaining shards will be loaded automatically.
226
+ // Addon logs per model (llamacpp, whispercpp, etc.)
227
+ for await (const log of loggingStream({ id: modelId })) {
228
+ console.log(`[${log.level}] ${log.namespace}: ${log.message}`);
229
+ }
230
+ ```
231
+
232
+ - Log streaming: [`examples/logging-streaming.ts`](examples/logging-streaming.ts)
233
+ - Log with custom file transport: [`examples/logging-file-transport.ts`](examples/logging-file-transport.ts)
234
+
235
+ ### Download Lifecycle
236
+
237
+ - Pause and resume download: [`examples/download-with-cancel.ts`](examples/download-with-cancel.ts)
303
238
 
304
239
  ### Blind Relays
305
240
 
@@ -308,17 +243,210 @@ Blind relays help establish peer connections through NAT/firewalls by routing tr
308
243
  - Model downloads via Hyperdrive: [`examples/download-with-blind-relays.ts`](./examples/download-with-blind-relays.ts)
309
244
  - Delegated inference: You can reuse the same pattern for delegated inference by calling `setConfig({ swarmRelays: [...relayPublicKeys] })` before starting your provider/consumer.
310
245
 
246
+ > [!NOTE]
247
+ > The examples use mock relay keys. For real deployments, you **must** use your own relay servers or trusted public relays.
248
+
249
+ ### Sharded Models
250
+
251
+ - Load sharded models from Hyperdrive: [`examples/llamacpp-sharded.ts`](examples/llamacpp-sharded.ts)
252
+ - Load models from local filesystem: [`examples/llamacpp-filesystem.ts`](examples/llamacpp-filesystem.ts)
253
+
254
+ > [!NOTE]
255
+ > Sharded models automatically download all parts sequentially with detailed progress tracking per shard. For local sharded models, pass the path to the first shard file (e.g., `model-00001-of-00005.gguf`), and remaining shards will be loaded automatically.
256
+
257
+ ## Basic flow
258
+
259
+ ```mermaid
260
+ sequenceDiagram
261
+ participant User as User
262
+ participant SDK as QVAC SDK
263
+ participant RPC as RPC Client (singleton)
264
+ participant Worker as Bare Worker (singleton)
265
+
266
+ User->>SDK: Call loadModel("llama-3", options)
267
+ SDK->>RPC: Create runtime-specific RPC client
268
+ RPC->>Worker: Spawn worker (if needed) and send loadModel request
269
+ Worker->>Worker: Download and load model into memory
270
+ Worker->>RPC: Return success response
271
+ RPC->>SDK: Model loaded successfully
272
+ SDK->>User: loadModel() resolves with modelId
273
+ ```
274
+
311
275
  **Note**: The example uses mock relay keys. In real deployments, you **must** use your own relay servers or trusted public relays.
312
276
 
313
- ## Known issues
277
+ ### Hot Config Reload
278
+
279
+ Hot config reload allows you to update model configurations on-the-fly without unloading the model. Pass `modelId` (instead of `modelSrc`) to `loadModel` with the `modelType` and new `modelConfig` to apply changes instantly.
280
+
281
+ - Config reload using whisper: [`examples/config-reload.ts`](examples/config-reload.ts)
282
+
283
+ **Note**: Config reload is currently supported for Whisper models. All config parameters except `contextParams` (GPU settings, flash attention) can be hot reloaded. `contextParams` are load-time only and require full model reload. More model types coming soon.
284
+
285
+ ## Contributing
286
+
287
+ ### Commit Message and PR Title Format
288
+
289
+ This repository enforces structured commit messages and PR titles to maintain consistency and generate changelogs automatically.
290
+
291
+ #### Format
292
+
293
+ **Commit messages:**
294
+
295
+ ```
296
+ prefix[tags]?: subject
297
+ ```
298
+
299
+ **PR titles:**
300
+
301
+ ```
302
+ TICKET prefix[tags]: subject
303
+ ```
304
+
305
+ #### Allowed Prefixes
306
+
307
+ - `feat` - New features or capabilities
308
+ - `fix` - Bug fixes
309
+ - `doc` - Documentation changes
310
+ - `test` - Test additions or modifications
311
+ - `mod` - Model-related changes
312
+ - `chore` - Maintenance tasks
313
+ - `infra` - CI/CD, tooling, infrastructure
314
+
315
+ #### Allowed Tags
316
+
317
+ Tags are optional:
318
+
319
+ - `[api]` - API changes (non-breaking)
320
+ - `[bc]` - Breaking changes (including breaking API changes)
321
+
322
+ #### Examples
323
+
324
+ **Valid commit messages:**
325
+
326
+ ```bash
327
+ feat: add RAG support for LanceDB
328
+ fix[api]: fix completion stream error handling
329
+ doc: update installation instructions
330
+ feat[bc]: redesign loadModel signature
331
+ chore: update dependencies
332
+ ```
333
+
334
+ **Valid PR titles:**
335
+
336
+ ```bash
337
+ QVAC-123 feat: add RAG support for LanceDB
338
+ QVAC-456 fix[api]: fix completion stream error handling
339
+ QVAC-789 doc: update installation instructions
340
+ QVAC-101 feat[bc]: redesign loadModel signature
341
+ ```
342
+
343
+ #### Code Examples Requirements
344
+
345
+ When creating PRs with specific tags, you must include code examples in the PR description:
346
+
347
+ **`[bc]` tag requirements:**
348
+
349
+ Must include BEFORE/AFTER code examples showing the migration path:
350
+
351
+ ````markdown
352
+ ## BC Changes
353
+
354
+ **BEFORE:**
355
+
356
+ ```typescript
357
+ const model = await loadModel("model-path");
358
+ ```
359
+
360
+ **AFTER:**
361
+
362
+ ```typescript
363
+ const modelId = await loadModel("model-path", { modelType: "llm" });
364
+ ```
365
+ ````
366
+
367
+ Or using inline comments:
368
+
369
+ ````markdown
370
+ ```typescript
371
+ // old
372
+ const model = await loadModel("model-path");
373
+
374
+ // new
375
+ const modelId = await loadModel("model-path", { modelType: "llm" });
376
+ ```
377
+ ````
378
+
379
+ **`[api]` tag requirements (non-breaking):**
380
+
381
+ Must include at least one fenced code block showing the new API usage:
382
+
383
+ ````markdown
384
+ ## New API
385
+
386
+ ```typescript
387
+ // New completion API with streaming support
388
+ for await (const token of completion({
389
+ modelId,
390
+ history: [{ role: "user", content: "Hello!" }],
391
+ }).tokenStream) {
392
+ process.stdout.write(token);
393
+ }
394
+ ```
395
+ ````
396
+
397
+ #### Validation
398
+
399
+ - **Commit messages** are validated automatically via Husky commit-msg hook
400
+ - **PR titles and descriptions** are validated via GitHub Actions on PR creation/update
401
+ - Invalid commits or PRs will be rejected with helpful error messages
402
+ - **Auto-skipped commits:** The following Git-generated commits bypass validation:
403
+ - Merge commits (e.g., `Merge pull request #123`)
404
+ - Version bumps (e.g., `1.0.0`, `v1.0.0`)
405
+ - Revert commits (e.g., `Revert "feat: add feature"`)
406
+ - Squash commits (e.g., `squash! fix: bug fix`)
407
+
408
+ #### Generating Changelogs
409
+
410
+ Once your PRs are merged into `dev`, you can generate a changelog:
411
+
412
+ ```bash
413
+ npm run changelog:generate
414
+ ```
415
+
416
+ This will:
417
+
418
+ 1. Compare versions between `dev` and `main` branches
419
+ 2. Collect all merged PRs
420
+ 3. Parse and classify each PR by prefix
421
+ 4. Generate `changelog/<version>/CHANGELOG.md`
422
+ 5. Generate `changelog/<version>/breaking.md` for BC changes (with code examples)
423
+ 6. Generate `changelog/<version>/api.md` for API changes (with code examples)
424
+
425
+ **Note:** Requires a GitHub token (`GITHUB_TOKEN` or `GH_TOKEN` environment variable) to fetch PR metadata.
426
+
427
+ ## Build
428
+
429
+ Use the [Bun](https://bun.sh/) package manager:
430
+
431
+ ```bash
432
+ bun i
433
+ ```
434
+
435
+ ```bash
436
+ bun run build # or `watch` for hotreload
437
+ ```
438
+
439
+ ```bash
440
+ bun run build:pack
441
+ ```
442
+
443
+ This outputs a tarball under `dist/qvac-sdk-{version}.tgz` that you can install in your project, e.g.:
444
+
445
+ ```bash
446
+ npm i path/to/qvac-sdk-0.3.0.tgz
447
+ ```
314
448
 
315
- - **One off scripts don't terminate / hang**
316
- - Add a `process.exit()` call at the end, we're working on it.
449
+ ## More Resources
317
450
 
318
- - **I'm building my Expo app on iOS but I don't see logs from QVAC**
319
- - `npx expo start`
320
- - Open the `ios` folder in Xcode
321
- - Start the app with the play ▶️ button
322
- - View -> Debug area -> Activate console
323
- - You should see the system logs on the right pane
324
- - ⚠️ This only works when starting the app from Xcode
451
+ - [Comprehensive documentation of this SDK](https://qvac.tether.dev/docs/sdk)
452
+ - [Package at NPM](https://www.npmjs.com/package/@qvac/sdk)