@qvac/llm-llamacpp 0.66.0 → 0.67.0

package/README.md

@@ -15,6 +15,7 @@ This native C++ addon, built using the `Bare` Runtime, simplifies running Large
   - [5. Load Model](#5-load-model)
   - [6. Run Inference](#6-run-inference)
     - [Tool calling](#tool-calling)
+    - [Ordered text and media](#ordered-text-and-media)
     - [Batch inference](#batch-inference)
   - [7. Release Resources](#7-release-resources)
 - [API behavior by state](#api-behavior-by-state)
@@ -206,7 +207,8 @@ const config = {
 | cache_checkpoints_max_bytes | integer (bytes)                 | 0 (unlimited)                | Byte budget for those checkpoints, enforced before the count. The load fails with `InvalidArgument` when it cannot hold `cache_checkpoints` checkpoints of the largest size the context allows (on a sliding-window model, a full window). Also accepted as `cache-checkpoints-max-bytes` |
 | cache_checkpoint_storage | `"memory"` or `"disk"`             | `"memory"`                   | Where checkpoints and the per-request rollback snapshot live. `"memory"` keeps them in host RAM so a cached chat never touches the disk; `"disk"` writes them to a private directory inside `cache_checkpoint_dir`, which it requires. Also accepted as `cache-checkpoint-storage` |
 | cache_checkpoint_dir | directory path                              | none                         | **Required with `cache_checkpoint_storage: "disk"`**, and refused without it: where checkpoint files go, inside a private directory the addon creates in it (mode 0700; on Windows, the directory itself). On mobile, pass the app's cache directory. The load fails with `InvalidArgument` when it is missing, empty, or unusable. Also accepted as `cache-checkpoint-dir` |
-| cache_ram_mib | 0 – 1048576 (MiB)                               | 0 (off)                      | Host-RAM budget for conversations that are not running. A `cacheKey` switch or a batch slot eviction moves the conversation there instead of writing its file, and the next request on that key restores it. A write-back cache: unsaved turns reach the file on `saveCache()`, when the budget pushes the entry out, or at unload; ephemeral entries are dropped instead. Also accepted as `cache-ram-mib` ([details](./docs/cache-api.md#keep-switched-away-conversations-in-ram)) |
+| cache_ram_mib | 0 – 1048576 (MiB)                               | 0 (off)                      | Host-RAM budget for conversations that are not running. A `cacheKey` switch or a batch slot eviction moves the conversation there instead of writing its file, and the next request on that key restores it; unless `cache_share_across_keys` is `"false"`, a request on another key that shares its prompt prefix starts from a copy of it. A write-back cache: unsaved turns reach the file on `saveCache()`, when the budget pushes the entry out, or at unload; ephemeral entries are dropped instead. Also accepted as `cache-ram-mib` ([details](./docs/cache-api.md#keep-switched-away-conversations-in-ram)) |
+| cache_share_across_keys | `"true"`/`"on"`/`"1"`, `"false"`/`"off"`/`"0"` | `"true"`                     | Whether a request on a decoder-only model may start from state another conversation left (keyless memory, another key's state, a `cacheKey` file the model used). `"false"` keeps every request to its own key's state, for callers who must not learn through `cacheTokens` or timing whether another conversation shares their prompt. Also accepted as `cache-share-across-keys` ([details](./docs/cache-api.md#keep-conversations-apart)) |
 | flash-attn        | `"on"`/`"enabled"`/`"true"`/`"1"`, `"off"`/`"disabled"`/`"false"`/`"0"`, or `"auto"` | `"on"`, except when finetuning or on a BitNet model, where it is forced off | Flash attention. The four truthy and four falsey spellings are equivalent; `"auto"` is a third state that defers to qvac-fabric's runtime capability probe. Lower-case only — matching is case-sensitive and any other value is rejected. Affects the KV-cache auto-default — see below. Also accepted as `flash_attn`; supplying both spellings is an error |
 | cache-type-k      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **key** quantization type. Unset = auto-default (see KV-cache type below) |
 | cache-type-v      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **value** quantization type. Quantizing V requires `flash-attn` on |
@@ -345,6 +347,28 @@ const response = await model.run([
 
 The tool grammar holds the model to the order of `parameters`. Inside an object it accepts the required properties first, in declared order, then the optional ones, also in declared order. When the model writes a property out of that order, the call isn't rejected with an error: the model drops a field, writes a broken value or never finishes the call. Declare `required` properties before optional ones at every level, and declare each group in the order the model is likely to write it. This applies to nested objects, and to every argument in formats that write the arguments as one JSON object (Qwen3). Qwen3.5's top-level `<parameter=...>` arguments are less strict: up to six required arguments in any order, then the optional ones in any order. An optional argument before a required one is still rejected there.
 
+#### Ordered text and media
+
+A user message's `content` is either a string or an ordered array of parts. Use the array when the position of an image inside the text matters — multiple-choice options that are pictures, illustrated forms, steps — so the model sees the text and every image in one turn, in order:
+
+```javascript
+const response = await model.run([{
+  role: 'user',
+  content: [
+    { type: 'text', text: 'Which image shows the same place as ' },
+    { type: 'media', content: '/images/place.jpg' },
+    { type: 'text', text: '?\nA. ' },
+    { type: 'media', content: imageABytes }, // raw bytes work too
+    { type: 'text', text: '\nB. ' },
+    { type: 'media', content: '/images/option-b.jpg' }
+  ]
+}])
+```
+
+Each media part becomes an `<__media__>` marker at its own position in the rendered turn — the same prompt llama-server builds from an OpenAI content-parts request. A part's `content` is an absolute path or a `Uint8Array`; media parts are only valid on user messages, and an unknown part type is rejected with `InvalidInputFormat`.
+
+The alternative — one `{ role: 'user', type: 'media' }` message per part — keeps the order but renders each text part as its own user turn, so text that refers to "this image" ends up in a different turn from the image.
+
 #### Batch inference
 
 Load the model with `parallel >= 2`, then pass an array of prompts. Each chunk arrives tagged with the id of the sequence that produced it; `await()` returns results in input order.
@@ -443,7 +467,7 @@ Admission is controlled by `rejectWhenBusy` (instance-level `opts.rejectWhenBusy
 
 #### Prefill (cache warming) with `parallel >= 2`
 
-A prefill-only run (`runOptions.prefill: true`) is admitted on a parallel model only when its product survives the slot teardown, i.e. it has a `cacheKey`: the warmed conversation stays in its slot for the next request on that key. A keyless prefill warms context state that no concurrent job could ever reach, so it is rejected with `InvalidArgument`; run it on a `parallel: 1` model instead. The same rule applies per item in batch runs. See [cache-api.md](./docs/cache-api.md).
+A prefill-only run (`runOptions.prefill: true`) is admitted on a parallel model only when its product survives the slot teardown, i.e. it has a `cacheKey`: the warmed conversation stays in its slot for the next request on that key. A keyless prefill warms state no later request can count on (at most parked anonymously, first to be evicted), so it is rejected with `InvalidArgument`; run it on a `parallel: 1` model instead. The same rule applies per item in batch runs. See [cache-api.md](./docs/cache-api.md).
 
 #### Cancelling a batch
 
@@ -735,7 +759,7 @@ npm run test:cpp:models:ci   # exactly what .github/workflows/cpp-tests-llm.yml
                              # downloads; matches what CI exercises
 ```
 
-First-run downloads pull several GB from Hugging Face. Every fixture is SHA256-verified against a digest pinned in `scripts/download-unit-test-models.js`; mismatched or partial files are re-downloaded automatically. Set `HF_TOKEN` if a repo requires authentication. Override paths with env vars such as `SHARDED_MODEL_FIRST_SHARD_PATH` (see `test/unit/test_common.hpp`).
+First-run downloads pull several GB, mostly from Hugging Face. Every fixture is SHA256-verified against a digest pinned in `scripts/download-unit-test-models.js`; mismatched or partial files are re-downloaded automatically. Set `HF_TOKEN` if a repo requires authentication. Fixtures pinned by `s3Path` (currently `bitnet_b1_58-large-TQ2_0.gguf`) are copied from the QVAC model registry bucket and need `MODEL_S3_BUCKET` plus AWS credentials; if either is missing or the copy fails, a local run skips the fixture with a warning and its tests skip, while `--ci` fails. Override paths with env vars such as `SHARDED_MODEL_FIRST_SHARD_PATH` or `BITNET_MODEL_PATH` (see `test/unit/test_common.hpp`).
 
 ### Fuzzing
 

package/addon.d.ts

@@ -14,6 +14,7 @@ export interface AddonMessage {
     generationParams?: GenerationParams;
     cacheKey?: string;
     ephemeral?: boolean;
+    baseCacheKey?: string;
 }
 export interface AddonMediaMessage {
     type: "media";

package/index.d.ts

@@ -380,6 +380,19 @@ declare namespace LlmLlamacpp {
          * maximum is 1048576 (1 TiB). Also accepted as `cache-ram-mib`.
          */
         cache_ram_mib?: NumericLike;
+        /**
+         * Whether a request on a decoder-only model may start from state another
+         * conversation left: the memory a request without a `cacheKey` keeps,
+         * another key's state in a sequence or the RAM tier, or a `cacheKey` file
+         * the model has used. `'true'` (default) shares it, as llama-server's
+         * prompt cache does. `'false'` keeps every request to its own key's state,
+         * for a model serving callers who must not learn through `cacheTokens` or
+         * timing whether another conversation shares their prompt: a request
+         * without a key starts cold and is reset afterwards, a new key starts cold,
+         * and `baseCacheKey` has no effect. Also accepted as
+         * `cache-share-across-keys`.
+         */
+        cache_share_across_keys?: "true" | "false" | "on" | "off" | "1" | "0";
         /**
          * Speculative decoding, as llama-server's `--spec-type`. `'draft-mtp'`
          * drafts with the model's own multi-token-prediction (MTP / NextN) head,
@@ -437,9 +450,30 @@ declare namespace LlmLlamacpp {
             rejectWhenBusy?: boolean;
         };
     }
+    /**
+     * One ordered piece of a message's `content`. Media parts keep their place in
+     * the turn: a single user message can interleave text and media, which a
+     * string `content` plus separate `type: "media"` messages cannot express
+     * without splitting the turn.
+     */
+    type UserContentPart = {
+        type: "text";
+        text: string;
+    } | {
+        type: "media";
+        /**
+         * Either an absolute path to the media file (`string`) or its raw bytes
+         * (`Uint8Array`), the same two forms a `type: "media"` message accepts.
+         */
+        content: string | Uint8Array;
+    };
     interface UserTextMessage {
         role: "system" | "assistant" | "user" | "tool" | "session" | string;
-        content: string;
+        /**
+         * Plain text, or ordered text/media parts rendered as one turn. Media parts
+         * are only valid on user messages.
+         */
+        content: string | UserContentPart[];
         /**
          * An assistant turn's reasoning, separate from its answer in `content`, as
          * in OpenAI-compatible APIs. Optional: an answer sent back with its
@@ -554,10 +588,9 @@ declare namespace LlmLlamacpp {
          * Run prefill only (cache warming): the prompt is evaluated but no tokens
          * are generated. On a model loaded with `parallel >= 2` a prefill needs a
          * `cacheKey` (the warmed conversation stays in its slot for the next
-         * request on that key) — a keyless prefill warms context state that no
-         * concurrent job could reach and is rejected with `InvalidArgument`; run
-         * keyless prefills on a `parallel: 1` model. The same rule applies per
-         * batch item.
+         * request on that key) — a keyless prefill warms state no later request
+         * can count on and is rejected with `InvalidArgument`; run keyless
+         * prefills on a `parallel: 1` model. The same rule applies per batch item.
          */
         prefill?: boolean;
         generationParams?: GenerationParams;
@@ -574,11 +607,26 @@ declare namespace LlmLlamacpp {
          * otherwise be lost with turns the file lacks: when it is set aside for
          * another key, evicted from its batch slot or from the RAM tier, or the
          * model is reloaded or unloaded. With `ephemeral: true` each of those
-         * drops it instead, so its next request starts cold. It still loads an
-         * existing file, and an explicit `saveCache(cacheKey)` still writes it.
+         * drops it instead, so its next request finds no state of its own (it
+         * still reuses any prefix it shares with what the model holds, except on
+         * encoder-decoder models). It still loads an existing file, and an
+         * explicit `saveCache(cacheKey)` still writes it.
          * The flag belongs to the conversation's latest request.
          */
         ephemeral?: boolean;
+        /**
+         * A `cacheKey` file to start from when nothing in memory serves more of
+         * the prompt, such as a system prompt and tools primed with `prefill`.
+         * It is read, never written: the request's own state goes to its
+         * `cacheKey`. A file that is missing or shares no prefix with the prompt
+         * is skipped. Once named, the file stays a source for later requests on
+         * this model, as any `cacheKey` file it used does, until files named
+         * or picked as a prompt's closest match more recently push it out (the
+         * model keeps the 256 most recent, as far as 64 MiB of memory holds
+         * their prompt records). No effect with `cache_share_across_keys:
+         * 'false'`.
+         */
+        baseCacheKey?: string;
         /**
          * Admission policy when the worker pool is full. `true` rejects before
          * submitting with an `Error` carrying `code === 'RUN_BUSY'` — branch on the

package/index.js

@@ -16,6 +16,7 @@ function normalizeRunOptions(runOptions) {
             generationParams: undefined,
             cacheKey: undefined,
             ephemeral: false,
+            baseCacheKey: undefined,
             rejectWhenBusy: undefined,
         };
     }
@@ -43,6 +44,9 @@ function normalizeRunOptions(runOptions) {
     if (options.ephemeral !== undefined && typeof options.ephemeral !== "boolean") {
         throw new TypeError("ephemeral must be a boolean when provided");
     }
+    if (options.baseCacheKey !== undefined && typeof options.baseCacheKey !== "string") {
+        throw new TypeError("baseCacheKey must be a string when provided");
+    }
     if (options.rejectWhenBusy !== undefined && typeof options.rejectWhenBusy !== "boolean") {
         throw new TypeError("rejectWhenBusy must be a boolean when provided");
     }
@@ -51,15 +55,28 @@ function normalizeRunOptions(runOptions) {
         generationParams: normalizeGenerationParams(options.generationParams),
         cacheKey: options.cacheKey,
         ephemeral: options.ephemeral === true,
+        baseCacheKey: options.baseCacheKey,
         // Left undefined when unset so admission falls back to the instance default.
         rejectWhenBusy: options.rejectWhenBusy,
     };
 }
+// Malformed parts (null, non-objects) are not byte media: they pass through
+// untouched so the native validator rejects them with `InvalidInputFormat`.
+function isByteMediaPart(part) {
+    const candidate = part;
+    return (candidate !== null &&
+        typeof candidate === "object" &&
+        candidate.type === "media" &&
+        candidate.content instanceof Uint8Array);
+}
 function promptToAddonMessages(prompt, runOptions) {
     if (!Array.isArray(prompt)) {
         throw new TypeError("Prompt input must be Message[]");
     }
-    const { prefill, generationParams, cacheKey, ephemeral } = normalizeRunOptions(runOptions);
+    const { prefill, generationParams, cacheKey, ephemeral, baseCacheKey } = normalizeRunOptions(runOptions);
+    // Only ever JSON-serialized into the addon's text message: a byte-buffer
+    // media part reaches the native side path-less, bound to the hoisted buffer
+    // by position, so the wire shape is wider than the public `Message` type.
     const textMessages = [];
     const mediaItems = [];
     for (const message of prompt) {
@@ -67,10 +84,26 @@ function promptToAddonMessages(prompt, runOptions) {
         if (media.role === "user" && media.type === "media" && media.content instanceof Uint8Array) {
             mediaItems.push(media.content);
             textMessages.push({ ...media, content: "" });
+            continue;
         }
-        else {
-            textMessages.push(message);
+        // Ordered parts: hoist each media part's bytes next to the other media
+        // messages and leave a path-less part behind. The native side records the
+        // plan in marker order, so the bytes still bind to the right markers.
+        const text = message;
+        if (Array.isArray(text.content)) {
+            textMessages.push({
+                ...text,
+                content: text.content.map((part) => {
+                    if (isByteMediaPart(part)) {
+                        mediaItems.push(part.content);
+                        return { type: "media" };
+                    }
+                    return part;
+                }),
+            });
+            continue;
         }
+        textMessages.push(message);
     }
     const promptMessages = [];
     for (const mediaData of mediaItems) {
@@ -83,6 +116,7 @@ function promptToAddonMessages(prompt, runOptions) {
         generationParams,
         cacheKey,
         ephemeral,
+        baseCacheKey,
     });
     return promptMessages;
 }
@@ -257,6 +291,15 @@ function sanitizePromptForLog(prompt) {
         if (media && media.type === "media" && media.content instanceof Uint8Array) {
             return { ...media, content: `[Uint8Array byteLength=${media.content.byteLength}]` };
         }
+        const text = msg;
+        if (Array.isArray(text?.content)) {
+            return {
+                ...text,
+                content: text.content.map((part) => isByteMediaPart(part)
+                    ? { ...part, content: `[Uint8Array byteLength=${part.content.byteLength}]` }
+                    : part),
+            };
+        }
         return msg;
     });
 }
@@ -863,9 +906,20 @@ const LlmLlamacpp = class LlmLlamacpp {
         if (text !== undefined)
             params.text = text;
         // Media bytes are dropped rather than serialized: the native side rejects
-        // any media message by its type.
+        // any media message or ordered media part by its type.
         if (history !== undefined) {
-            params.history = JSON.stringify(history.map((message) => message?.type === "media" ? { ...message, content: "" } : message));
+            params.history = JSON.stringify(history.map((message) => {
+                if (message?.type === "media")
+                    return { ...message, content: "" };
+                const textMessage = message;
+                if (Array.isArray(textMessage?.content)) {
+                    return {
+                        ...message,
+                        content: textMessage.content.map((part) => isByteMediaPart(part) ? { type: "media" } : part),
+                    };
+                }
+                return message;
+            }));
         }
         if (tools !== undefined)
             params.tools = JSON.stringify(tools);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@qvac/llm-llamacpp",
-  "version": "0.66.0",
+  "version": "0.67.0",
   "description": "llama addon for qvac",
   "addon": true,
   "scripts": {