kjarni-wasm 0.1.4 → 0.1.5

package/README.md

@@ -1,7 +1,7 @@
 # kjarni-wasm
 
-Local AI inference in the browser. Embeddings, classification, reranking and chat,
-running in a Web Worker so the page never freezes.
+Local AI inference in the browser. Embeddings, classification, reranking, semantic
+search and chat, running in a Web Worker so the page never freezes.
 
 ```bash
 npm i kjarni-wasm
@@ -20,7 +20,9 @@ Or without a bundler, straight from a CDN:
 ```ts
 import { Kjarni } from "kjarni-wasm";
 
-const kjarni = await Kjarni.load({ encoder: "/models/minilm-l6-v2-q8.kjq" });
+const kjarni = await Kjarni.load({
+  encoder: "https://huggingface.co/olafuraron/all-MiniLM-L6-v2-q8/resolve/main/all-MiniLM-L6-v2-q8.kjq",
+});
 
 const [doctor, physician] = await kjarni.encode(["doctor", "physician"]);
 console.log(Kjarni.cosine(doctor, physician));   // 0.8598
@@ -29,6 +31,8 @@ console.log(Kjarni.cosine(doctor, physician));   // 0.8598
 console.log(await kjarni.similarity("doctor", "physician"));
 ```
 
+That URL is a real, public 23 MB file. It downloads once, the browser caches it, and nothing else leaves the machine. See [Models](#models) for the rest, and serve them from your own origin in production rather than hotlinking.
+
 Every method returns a promise and the work happens on a worker thread. A model
 loads or a reply generates while the page keeps painting.
 
@@ -48,9 +52,9 @@ Models are `.kjq` files: config, tokenizer and int8 weights in one download.
 
 ```ts
 const kjarni = await Kjarni.load({
-  encoder:    "/models/minilm-l6-v2-q8.kjq",
-  classifier: "/models/distilbert-sentiment-q8.kjq",
-  reranker:   "/models/ms-marco-MiniLM-L-6-v2-q8.kjq",
+  encoder:    model("all-MiniLM-L6-v2-q8"),
+  classifier: model("distilbert-sentiment-q8"),
+  reranker:   model("ms-marco-MiniLM-L-6-v2-q8"),
   onProgress: (capability, loaded, total) => {
     console.log(`${capability}: ${Math.round((loaded / total) * 100)}%`);
   },
@@ -85,22 +89,32 @@ for (const { index, score } of ranked) console.log(score, documents[index]);
 
 ## Chat
 
+A 0.5B language model runs in the tab, and tokens arrive as they are decoded.
+
 ```ts
 const kjarni = await Kjarni.load({
-  chat: "/models/qwen05b-q8.kjq",
+  chat: model("qwen05b-q8"),
   chatModelId: "qwen2.5-0.5b-instruct",
 });
 
-for await (const piece of kjarni.chat("Explain RAG in one sentence.")) {
-  output.textContent += piece;
-}
+let out = "";
+for await (const piece of kjarni.chat("Explain RAG in one sentence.")) out += piece;
 ```
 
-Tokens arrive as they are decoded, not in one lump at the end. The worker is what
-makes that useful: generation occupies its thread completely, so the same call on
-the main thread would freeze the tab until the reply finished.
+This works because the weights stay block-quantised in memory. wasm32 caps a
+single allocation at `isize::MAX`, which is 2 GB, and Qwen2.5 0.5B expanded to f32
+needs 1.98 GB, so a loader that dequantises up front traps on load. The `.kjq`
+container carries block-quantised weights the engine reads in place, which is
+500 MB rather than 1.98 GB, and decodes faster than f32 rather than slower.
+
+Expect a few tokens per second on a laptop CPU. Run it in a Web Worker: generation
+occupies the thread until it returns, so on the main thread the tab stops painting
+for the whole reply.
 
-`temperature: 0` selects greedy decoding, which is deterministic.
+`chat()` completes a prompt; it does not keep turns. Multi-turn conversation, where
+the model's own chat template is applied and earlier turns are remembered, exists
+on `WasmChat` (`send_stream`, `clear_history`) and is not yet surfaced through this
+wrapper.
 
 ## Cleaning up
 
@@ -116,6 +130,45 @@ reference leaks the weights. `close()` frees the models and stops the worker.
 A browser with WebAssembly and module workers: Chrome and Edge 80+, Firefox 114+,
 Safari 15+. No WebGPU required, inference runs on the CPU.
 
+## Models
+
+`.kjq` is a single file holding config, tokenizer and int8 weights. These are
+public and hosted on Hugging Face, one repository per model, with the repository
+and file names matching:
+
+```ts
+const model = (name: string) =>
+  `https://huggingface.co/olafuraron/${name}/resolve/main/${name}.kjq`;
+```
+
+| Pass as | Name | Size | What it does |
+| ------- | ---- | ---- | ------------ |
+| `encoder` | `all-MiniLM-L6-v2-q8` | 23 MB | embeddings, similarity, search |
+| `reranker` | `ms-marco-MiniLM-L-6-v2-q8` | 23 MB | cross-encoder reranking |
+| `classifier` | `distilbert-sentiment-q8` | 68 MB | positive / negative sentiment |
+| `chat` | `qwen05b-q8` | 508 MB | a local language model, block-quantised so it fits |
+
+Hugging Face serves these with permissive CORS, so a browser can fetch them from
+any origin. For production, copy what you need and serve it from your own origin:
+you control caching, and your page does not depend on someone else's uptime.
+
+Load only what you use, and wire `onProgress` to something visible: these are real
+downloads, and a silent minute reads as a broken page.
+
+## The same engine elsewhere
+
+Kjarni is one Rust engine behind several packages. The browser is one target; if
+you need the server half of the same product:
+
+- **[Live demo](https://kjarni.ai/demo/)**: this package, running in a page
+- **[Kjarni](https://www.nuget.org/packages/Kjarni)** and
+  **[Kjarni.Extensions.AI](https://www.nuget.org/packages/Kjarni.Extensions.AI)**
+  on NuGet for C#, including `IEmbeddingGenerator` and `IChatClient`
+- **[Go](https://pkg.go.dev/github.com/olafurjohannsson/kjarni-go)**,
+  **[C++](https://github.com/olafurjohannsson/kjarni/tree/main/crates/kjarni-ffi/examples/cpp)**
+  and a CLI that reads stdin and writes JSON
+- **[Source and issues](https://github.com/olafurjohannsson/kjarni)**
+
 ## Making your own `.kjq`
 
 See [the format guide](../scripts/README.md). Any BERT-family encoder,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "kjarni-wasm",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "description": "Local AI inference in the browser: embeddings, classification, reranking, semantic search and chat, running in a Web Worker.",
   "type": "module",
   "main": "./dist/index.js",

package/pkg/kjarni_wasm.d.ts

@@ -18,6 +18,10 @@ export class WasmChat {
     private constructor();
     free(): void;
     [Symbol.dispose](): void;
+    /**
+     * Drops the conversation, keeping the system prompt.
+     */
+    clear_history(): void;
     /**
      * The model's context window, in tokens.
      */
@@ -41,6 +45,10 @@ export class WasmChat {
      * itself is in `generate_stream_core`, which builds and is tested natively.
      */
     generate_stream(prompt: string, max_new_tokens: number, temperature: number, on_token: Function): string;
+    /**
+     * Number of messages held, system prompt included.
+     */
+    history_len(): number;
     /**
      * Load a decoder model from a `.kjq` container.
      *
@@ -48,6 +56,18 @@ export class WasmChat {
      * one the model still generates, but its turn markers will not be applied.
      */
     static load(data: Uint8Array, model_id?: string | null): WasmChat;
+    /**
+     * Sends a message in an ongoing conversation, streaming the reply.
+     *
+     * Unlike `generate_stream`, which completes raw text, this keeps the turns so
+     * far and formats them with the model's own chat template, so an instruct
+     * model answers instead of continuing the sentence. The reply is appended to
+     * the history, so the next call sees it.
+     *
+     * Browser-only: `js_sys::Function` has no native equivalent. The work is in
+     * `send_stream_core`, which builds and is tested natively.
+     */
+    send_stream(message: string, max_new_tokens: number, temperature: number, on_token: Function): string;
 }
 
 /**
@@ -190,10 +210,13 @@ export interface InitOutput {
     readonly __wbg_wasmindexbuilder_free: (a: number, b: number) => void;
     readonly __wbg_wasmmodel_free: (a: number, b: number) => void;
     readonly set_debug_logging: (a: number) => void;
+    readonly wasmchat_clear_history: (a: number) => void;
     readonly wasmchat_context_size: (a: number) => number;
     readonly wasmchat_generate: (a: number, b: number, c: number, d: number, e: number, f: number) => void;
     readonly wasmchat_generate_stream: (a: number, b: number, c: number, d: number, e: number, f: number, g: number) => void;
+    readonly wasmchat_history_len: (a: number) => number;
     readonly wasmchat_load: (a: number, b: number, c: number, d: number, e: number) => void;
+    readonly wasmchat_send_stream: (a: number, b: number, c: number, d: number, e: number, f: number, g: number) => void;
     readonly wasmclassifier_classify: (a: number, b: number, c: number, d: number) => void;
     readonly wasmclassifier_classify_batch: (a: number, b: number, c: number, d: number) => void;
     readonly wasmclassifier_labels: (a: number, b: number) => void;
@@ -225,8 +248,8 @@ export interface InitOutput {
     readonly wasmsearch_add_chunk: (a: number, b: number, c: number, d: number, e: number, f: number, g: number, h: number, i: number) => void;
     readonly __wbg_wasmreranker_free: (a: number, b: number) => void;
     readonly __wbg_wasmsearch_free: (a: number, b: number) => void;
-    readonly __wasm_bindgen_func_elem_5410: (a: number, b: number, c: number, d: number) => void;
-    readonly __wasm_bindgen_func_elem_5426: (a: number, b: number, c: number, d: number) => void;
+    readonly __wasm_bindgen_func_elem_5449: (a: number, b: number, c: number, d: number) => void;
+    readonly __wasm_bindgen_func_elem_5465: (a: number, b: number, c: number, d: number) => void;
     readonly __wbindgen_export: (a: number, b: number) => number;
     readonly __wbindgen_export2: (a: number, b: number, c: number, d: number) => number;
     readonly __wbindgen_export3: (a: number) => void;

package/pkg/kjarni_wasm.js

@@ -30,6 +30,12 @@ export class WasmChat {
         const ptr = this.__destroy_into_raw();
         wasm.__wbg_wasmchat_free(ptr, 0);
     }
+    /**
+     * Drops the conversation, keeping the system prompt.
+     */
+    clear_history() {
+        wasm.wasmchat_clear_history(this.__wbg_ptr);
+    }
     /**
      * The model's context window, in tokens.
      * @returns {number}
@@ -117,6 +123,14 @@ export class WasmChat {
             wasm.__wbindgen_export4(deferred3_0, deferred3_1, 1);
         }
     }
+    /**
+     * Number of messages held, system prompt included.
+     * @returns {number}
+     */
+    history_len() {
+        const ret = wasm.wasmchat_history_len(this.__wbg_ptr);
+        return ret >>> 0;
+    }
     /**
      * Load a decoder model from a `.kjq` container.
      *
@@ -145,6 +159,49 @@ export class WasmChat {
             wasm.__wbindgen_add_to_stack_pointer(16);
         }
     }
+    /**
+     * Sends a message in an ongoing conversation, streaming the reply.
+     *
+     * Unlike `generate_stream`, which completes raw text, this keeps the turns so
+     * far and formats them with the model's own chat template, so an instruct
+     * model answers instead of continuing the sentence. The reply is appended to
+     * the history, so the next call sees it.
+     *
+     * Browser-only: `js_sys::Function` has no native equivalent. The work is in
+     * `send_stream_core`, which builds and is tested natively.
+     * @param {string} message
+     * @param {number} max_new_tokens
+     * @param {number} temperature
+     * @param {Function} on_token
+     * @returns {string}
+     */
+    send_stream(message, max_new_tokens, temperature, on_token) {
+        let deferred3_0;
+        let deferred3_1;
+        try {
+            const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
+            const ptr0 = passStringToWasm0(message, wasm.__wbindgen_export, wasm.__wbindgen_export2);
+            const len0 = WASM_VECTOR_LEN;
+            wasm.wasmchat_send_stream(retptr, this.__wbg_ptr, ptr0, len0, max_new_tokens, temperature, addBorrowedObject(on_token));
+            var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
+            var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
+            var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
+            var r3 = getDataViewMemory0().getInt32(retptr + 4 * 3, true);
+            var ptr2 = r0;
+            var len2 = r1;
+            if (r3) {
+                ptr2 = 0; len2 = 0;
+                throw takeObject(r2);
+            }
+            deferred3_0 = ptr2;
+            deferred3_1 = len2;
+            return getStringFromWasm0(ptr2, len2);
+        } finally {
+            wasm.__wbindgen_add_to_stack_pointer(16);
+            heap[stack_pointer++] = undefined;
+            wasm.__wbindgen_export4(deferred3_0, deferred3_1, 1);
+        }
+    }
 }
 if (Symbol.dispose) WasmChat.prototype[Symbol.dispose] = WasmChat.prototype.free;
 
@@ -1035,7 +1092,7 @@ function __wbg_get_imports() {
                     const a = state0.a;
                     state0.a = 0;
                     try {
-                        return __wasm_bindgen_func_elem_5426(a, state0.b, arg0, arg1);
+                        return __wasm_bindgen_func_elem_5465(a, state0.b, arg0, arg1);
                     } finally {
                         state0.a = a;
                     }
@@ -1058,6 +1115,14 @@ function __wbg_get_imports() {
             const ret = Date.now();
             return ret;
         },
+        __wbg_now_e7c6795a7f81e10f: function(arg0) {
+            const ret = getObject(arg0).now();
+            return ret;
+        },
+        __wbg_performance_3fcf6e32a7e1ed0a: function(arg0) {
+            const ret = getObject(arg0).performance;
+            return addHeapObject(ret);
+        },
         __wbg_process_44c7a14e11e9f69e: function(arg0) {
             const ret = getObject(arg0).process;
             return addHeapObject(ret);
@@ -1141,8 +1206,8 @@ function __wbg_get_imports() {
             return addHeapObject(ret);
         },
         __wbindgen_cast_0000000000000001: function(arg0, arg1) {
-            // Cast intrinsic for `Closure(Closure { owned: true, function: Function { arguments: [Externref], shim_idx: 2036, ret: Result(Unit), inner_ret: Some(Result(Unit)) }, mutable: true }) -> Externref`.
-            const ret = makeMutClosure(arg0, arg1, __wasm_bindgen_func_elem_5410);
+            // Cast intrinsic for `Closure(Closure { owned: true, function: Function { arguments: [Externref], shim_idx: 2043, ret: Result(Unit), inner_ret: Some(Result(Unit)) }, mutable: true }) -> Externref`.
+            const ret = makeMutClosure(arg0, arg1, __wasm_bindgen_func_elem_5449);
             return addHeapObject(ret);
         },
         __wbindgen_cast_0000000000000002: function(arg0) {
@@ -1179,10 +1244,10 @@ function __wbg_get_imports() {
     };
 }
 
-function __wasm_bindgen_func_elem_5410(arg0, arg1, arg2) {
+function __wasm_bindgen_func_elem_5449(arg0, arg1, arg2) {
     try {
         const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
-        wasm.__wasm_bindgen_func_elem_5410(retptr, arg0, arg1, addHeapObject(arg2));
+        wasm.__wasm_bindgen_func_elem_5449(retptr, arg0, arg1, addHeapObject(arg2));
         var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
         var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
         if (r1) {
@@ -1193,8 +1258,8 @@ function __wasm_bindgen_func_elem_5410(arg0, arg1, arg2) {
     }
 }
 
-function __wasm_bindgen_func_elem_5426(arg0, arg1, arg2, arg3) {
-    wasm.__wasm_bindgen_func_elem_5426(arg0, arg1, addHeapObject(arg2), addHeapObject(arg3));
+function __wasm_bindgen_func_elem_5465(arg0, arg1, arg2, arg3) {
+    wasm.__wasm_bindgen_func_elem_5465(arg0, arg1, addHeapObject(arg2), addHeapObject(arg3));
 }
 
 const WasmChatFinalization = (typeof FinalizationRegistry === 'undefined')

package/pkg/kjarni_wasm_bg.wasm.d.ts

@@ -7,10 +7,13 @@ export const __wbg_wasmencoder_free: (a: number, b: number) => void;
 export const __wbg_wasmindexbuilder_free: (a: number, b: number) => void;
 export const __wbg_wasmmodel_free: (a: number, b: number) => void;
 export const set_debug_logging: (a: number) => void;
+export const wasmchat_clear_history: (a: number) => void;
 export const wasmchat_context_size: (a: number) => number;
 export const wasmchat_generate: (a: number, b: number, c: number, d: number, e: number, f: number) => void;
 export const wasmchat_generate_stream: (a: number, b: number, c: number, d: number, e: number, f: number, g: number) => void;
+export const wasmchat_history_len: (a: number) => number;
 export const wasmchat_load: (a: number, b: number, c: number, d: number, e: number) => void;
+export const wasmchat_send_stream: (a: number, b: number, c: number, d: number, e: number, f: number, g: number) => void;
 export const wasmclassifier_classify: (a: number, b: number, c: number, d: number) => void;
 export const wasmclassifier_classify_batch: (a: number, b: number, c: number, d: number) => void;
 export const wasmclassifier_labels: (a: number, b: number) => void;
@@ -42,8 +45,8 @@ export const wasmsearch_doc_count: (a: number) => number;
 export const wasmsearch_add_chunk: (a: number, b: number, c: number, d: number, e: number, f: number, g: number, h: number, i: number) => void;
 export const __wbg_wasmreranker_free: (a: number, b: number) => void;
 export const __wbg_wasmsearch_free: (a: number, b: number) => void;
-export const __wasm_bindgen_func_elem_5410: (a: number, b: number, c: number, d: number) => void;
-export const __wasm_bindgen_func_elem_5426: (a: number, b: number, c: number, d: number) => void;
+export const __wasm_bindgen_func_elem_5449: (a: number, b: number, c: number, d: number) => void;
+export const __wasm_bindgen_func_elem_5465: (a: number, b: number, c: number, d: number) => void;
 export const __wbindgen_export: (a: number, b: number) => number;
 export const __wbindgen_export2: (a: number, b: number, c: number, d: number) => number;
 export const __wbindgen_export3: (a: number) => void;