@qvac/llm-llamacpp 0.65.1 → 0.67.0

package/README.md

@@ -15,6 +15,7 @@ This native C++ addon, built using the `Bare` Runtime, simplifies running Large
   - [5. Load Model](#5-load-model)
   - [6. Run Inference](#6-run-inference)
     - [Tool calling](#tool-calling)
+    - [Ordered text and media](#ordered-text-and-media)
     - [Batch inference](#batch-inference)
   - [7. Release Resources](#7-release-resources)
 - [API behavior by state](#api-behavior-by-state)
@@ -206,7 +207,8 @@ const config = {
 | cache_checkpoints_max_bytes | integer (bytes)                 | 0 (unlimited)                | Byte budget for those checkpoints, enforced before the count. The load fails with `InvalidArgument` when it cannot hold `cache_checkpoints` checkpoints of the largest size the context allows (on a sliding-window model, a full window). Also accepted as `cache-checkpoints-max-bytes` |
 | cache_checkpoint_storage | `"memory"` or `"disk"`             | `"memory"`                   | Where checkpoints and the per-request rollback snapshot live. `"memory"` keeps them in host RAM so a cached chat never touches the disk; `"disk"` writes them to a private directory inside `cache_checkpoint_dir`, which it requires. Also accepted as `cache-checkpoint-storage` |
 | cache_checkpoint_dir | directory path                              | none                         | **Required with `cache_checkpoint_storage: "disk"`**, and refused without it: where checkpoint files go, inside a private directory the addon creates in it (mode 0700; on Windows, the directory itself). On mobile, pass the app's cache directory. The load fails with `InvalidArgument` when it is missing, empty, or unusable. Also accepted as `cache-checkpoint-dir` |
-| cache_ram_mib | 0 – 1048576 (MiB)                               | 0 (off)                      | Host-RAM budget for conversations that are not running. A `cacheKey` switch or a batch slot eviction moves the conversation there instead of writing its file, and the next request on that key restores it. A write-back cache: unsaved turns reach the file on `saveCache()`, when the budget pushes the entry out, or at unload; ephemeral entries are dropped instead. Also accepted as `cache-ram-mib` ([details](./docs/cache-api.md#keep-switched-away-conversations-in-ram)) |
+| cache_ram_mib | 0 – 1048576 (MiB)                               | 0 (off)                      | Host-RAM budget for conversations that are not running. A `cacheKey` switch or a batch slot eviction moves the conversation there instead of writing its file, and the next request on that key restores it; unless `cache_share_across_keys` is `"false"`, a request on another key that shares its prompt prefix starts from a copy of it. A write-back cache: unsaved turns reach the file on `saveCache()`, when the budget pushes the entry out, or at unload; ephemeral entries are dropped instead. Also accepted as `cache-ram-mib` ([details](./docs/cache-api.md#keep-switched-away-conversations-in-ram)) |
+| cache_share_across_keys | `"true"`/`"on"`/`"1"`, `"false"`/`"off"`/`"0"` | `"true"`                     | Whether a request on a decoder-only model may start from state another conversation left (keyless memory, another key's state, a `cacheKey` file the model used). `"false"` keeps every request to its own key's state, for callers who must not learn through `cacheTokens` or timing whether another conversation shares their prompt. Also accepted as `cache-share-across-keys` ([details](./docs/cache-api.md#keep-conversations-apart)) |
 | flash-attn        | `"on"`/`"enabled"`/`"true"`/`"1"`, `"off"`/`"disabled"`/`"false"`/`"0"`, or `"auto"` | `"on"`, except when finetuning or on a BitNet model, where it is forced off | Flash attention. The four truthy and four falsey spellings are equivalent; `"auto"` is a third state that defers to qvac-fabric's runtime capability probe. Lower-case only — matching is case-sensitive and any other value is rejected. Affects the KV-cache auto-default — see below. Also accepted as `flash_attn`; supplying both spellings is an error |
 | cache-type-k      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **key** quantization type. Unset = auto-default (see KV-cache type below) |
 | cache-type-v      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **value** quantization type. Quantizing V requires `flash-attn` on |
@@ -265,7 +267,7 @@ projector runs on CPU. The resolved choice is logged at verbosity ≥ 2 as
 
 `spec-type: 'draft-mtp'` turns on llama-server's multi-token-prediction speculative decoding for models that ship their MTP layers (`blk.N.nextn.*`, e.g. Qwen3.5 / Qwen3.6 GGUFs that keep them; quantizations that strip them fail to load with this option unless an external head is given, see below). Every step the MTP head drafts up to `spec-draft-n-max` tokens from the model's own hidden states, the model verifies the sampled token and the draft in one decode, and the longest matching prefix plus one new token is kept. The sampler decides every kept token, so sampling settings, grammars and stop conditions behave as without speculation. The implementation follows llama-server step by step: the same draft limits, rollback by trimming or by a checkpoint for memory that cannot trim, and the same replay.
 
-For a model without MTP layers, give an external MTP head as `spec-draft-model` (llama-server's `--spec-type draft-mtp -md <head>`): a GGUF with MTP layers for the same model, such as the MTP quantization of it, or a Gemma 4 assistant for its Gemma 4 model. The head is loaded next to the model and placed like it (`device`, `gpu-layers`), and the load's fit and `assessFit` count its weights as well as its context. A head without MTP layers fails the load, and `assessFit` refuses it with `unsupported-config`. A head for another model, with a different hidden width or vocabulary, also fails the load, and `assessFit` refuses it the same way. A Gemma 4 assistant reads the model's own tensors, so fabric measures it next to the model, as the load creates it.
+For a model without MTP layers, give an external MTP head as `spec-draft-model` (llama-server's `--spec-type draft-mtp -md <head>`): a GGUF with MTP layers for the same model, such as the MTP quantization of it, or a Gemma 4 assistant for its Gemma 4 model. The head is loaded next to the model and placed like it (`device`, `gpu-layers`), and the load's fit and `assessFit` count its weights as well as its context. A head without MTP layers fails the load, and `assessFit` rejects it. A head for another model, with a different hidden width or vocabulary, also fails the load, and `assessFit` rejects it the same way. A Gemma 4 assistant reads the model's own tensors, so fabric measures it next to the model, as the load creates it.
 
 `spec-type: 'draft-dflash'` drafts with a separate DFlash model instead, given as `spec-draft-model` (an absolute path; llama-server's `-md`). The DFlash model reads the hidden states of the target layers it was trained on and denoises a whole block of draft tokens in one pass, so a step drafts up to its block size minus one tokens (`spec-draft-n-max` caps it further). It is loaded next to the model and placed like it (`device`, `gpu-layers`). The load's fit and `assessFit` count its weights and context, measured next to the model when the drafter borrows the model's embeddings and output head. A draft model fabric still cannot measure is left out of both, and fabric logs `failed to measure the memory of the extra model, fitting without it`. Everything else (verification, rollback, batching, caches, stats) is the same as for MTP.
 
@@ -345,6 +347,28 @@ const response = await model.run([
 
 The tool grammar holds the model to the order of `parameters`. Inside an object it accepts the required properties first, in declared order, then the optional ones, also in declared order. When the model writes a property out of that order, the call isn't rejected with an error: the model drops a field, writes a broken value or never finishes the call. Declare `required` properties before optional ones at every level, and declare each group in the order the model is likely to write it. This applies to nested objects, and to every argument in formats that write the arguments as one JSON object (Qwen3). Qwen3.5's top-level `<parameter=...>` arguments are less strict: up to six required arguments in any order, then the optional ones in any order. An optional argument before a required one is still rejected there.
 
+#### Ordered text and media
+
+A user message's `content` is either a string or an ordered array of parts. Use the array when the position of an image inside the text matters — multiple-choice options that are pictures, illustrated forms, steps — so the model sees the text and every image in one turn, in order:
+
+```javascript
+const response = await model.run([{
+  role: 'user',
+  content: [
+    { type: 'text', text: 'Which image shows the same place as ' },
+    { type: 'media', content: '/images/place.jpg' },
+    { type: 'text', text: '?\nA. ' },
+    { type: 'media', content: imageABytes }, // raw bytes work too
+    { type: 'text', text: '\nB. ' },
+    { type: 'media', content: '/images/option-b.jpg' }
+  ]
+}])
+```
+
+Each media part becomes an `<__media__>` marker at its own position in the rendered turn — the same prompt llama-server builds from an OpenAI content-parts request. A part's `content` is an absolute path or a `Uint8Array`; media parts are only valid on user messages, and an unknown part type is rejected with `InvalidInputFormat`.
+
+The alternative — one `{ role: 'user', type: 'media' }` message per part — keeps the order but renders each text part as its own user turn, so text that refers to "this image" ends up in a different turn from the image.
+
 #### Batch inference
 
 Load the model with `parallel >= 2`, then pass an array of prompts. Each chunk arrives tagged with the id of the sequence that produced it; `await()` returns results in input order.
@@ -384,6 +408,30 @@ const response = await model.run([
 ])
 ```
 
+#### Counting tokens
+
+`calcTokens` returns how many tokens an input takes on the loaded model, without running it:
+
+```javascript
+const raw = await model.calcTokens({ text: 'Hello, world!' })
+
+const prompt = await model.calcTokens({
+  history: [{ role: 'system', content: 'You are a helpful assistant.' }],
+  tools: [{ type: 'function', name: 'get_weather', parameters: { type: 'object', properties: { city: { type: 'string' } } } }],
+  text: 'Weather in Paris?'
+})
+```
+
+- `text` alone is counted raw, with no chat template and no BOS.
+- `history` takes the `Message[]` that `run()` takes, and `tools` takes tool definitions. Either one is rendered by the chat template as the opening request of a fresh conversation, with `text` (even empty) appended as the last user turn. The count includes BOS, and the generation prompt when the last turn is a user or tool turn, so it equals the `promptTokens` a fresh `run()` of the same conversation reports. Tools the template drops are not counted. On an encoder-decoder model `run()` encodes the prompt separately, so its `promptTokens` differs from this count.
+- `tools` needs `text` or `history`, as `run()` can't render tools without a message.
+- It never queues behind `run()`, so it answers while requests are generating.
+- Errors:
+  - A wrong-typed argument rejects with a `TypeError`.
+  - Before `load()` or after `unload()` it rejects with `Model is not loaded`.
+  - Other invalid input, media included, rejects with an error whose `code` is `InvalidArgument` and whose message starts with `calcTokens:`.
+  - A conversation the chat template cannot render at all fails as it would in `run()`.
+
 ### 7. Release Resources
 
 Unload the model when finished:
@@ -398,7 +446,7 @@ try {
 
 ### API behavior by state
 
-The following table describes the expected behavior of `run` and `cancel` depending on the current state (idle vs jobs running). The two cancel entry points have different scopes:
+The following table describes the expected behavior of `run`, `cancel` and `calcTokens` depending on the current state (idle vs jobs running). The two cancel entry points have different scopes:
 
 - `response.cancel()` — **targeted**: cancels only the job (or batch group) that `run()` call produced, leaving other concurrent jobs running.
 - `model.cancel()` — **global**: cancels every live job (in-flight and queued) at the moment of the call, plus any finetuning in progress.
@@ -411,6 +459,7 @@ The following table describes the expected behavior of `run` and `cancel` depend
 | busy, `parallel >= 2` (default `rejectWhenBusy: false`) | run | **Allowed** — the job is admitted concurrently (continuous batching) or queued until a slot frees; each call gets its own independent `QvacResponse` |
 | busy          | `response.cancel()` | **Allowed** — cancels only that response's job/group; Promise resolves when it has stopped |
 | busy          | `model.cancel()`    | **Allowed** — cancels all live jobs; Promise resolves when they have stopped |
+| idle or busy  | `calcTokens` | **Allowed** — counts without running and never queues behind a job; see [Counting tokens](#counting-tokens) |
 
 Admission is controlled by `rejectWhenBusy` (instance-level `opts.rejectWhenBusy`, overridable per call via `runOptions.rejectWhenBusy`). Its default follows `parallel`: `true` for `parallel: 1` (busy runs fail fast, preserving the historical single-job contract) and `false` for `parallel >= 2` (busy runs queue behind the pool and start as slots free). With `parallel >= 2`, separate top-level `run()` calls are batched together into the same decode loop — see [Continuous Batching](./docs/continuous-batching.md).
 
@@ -418,7 +467,7 @@ Admission is controlled by `rejectWhenBusy` (instance-level `opts.rejectWhenBusy
 
 #### Prefill (cache warming) with `parallel >= 2`
 
-A prefill-only run (`runOptions.prefill: true`) is admitted on a parallel model only when its product survives the slot teardown, i.e. it has a `cacheKey`: the warmed conversation stays in its slot for the next request on that key. A keyless prefill warms context state that no concurrent job could ever reach, so it is rejected with `InvalidArgument`; run it on a `parallel: 1` model instead. The same rule applies per item in batch runs. See [cache-api.md](./docs/cache-api.md).
+A prefill-only run (`runOptions.prefill: true`) is admitted on a parallel model only when its product survives the slot teardown, i.e. it has a `cacheKey`: the warmed conversation stays in its slot for the next request on that key. A keyless prefill warms state no later request can count on (at most parked anonymously, first to be evicted), so it is rejected with `InvalidArgument`; run it on a `parallel: 1` model instead. The same rule applies per item in batch runs. See [cache-api.md](./docs/cache-api.md).
 
 #### Cancelling a batch
 
@@ -444,26 +493,33 @@ const fit = await LlmLlamacpp.assessFit({
   config: { device: 'gpu', 'ctx-size': '4096' }
 })
 
-fit.status // 'fits' | 'does-not-fit' | 'error'
-fit.reason // 'fits', 'does-not-fit', 'model-unreadable', 'no-backend-device' or 'unsupported-config'
-fit.gpuLayers // what fits, which is not always what the request asked for
-fit.ctxSize
-fit.devices // one row per device the model was assigned to, then a `host` row
-fit.deviceBytes // model, context and compute summed across the devices, host excluded
-fit.hostBytes // the same, for the trailing host row
+fit.status // 'fits' | 'does-not-fit' | 'unknown'
+fit.devices // one row per device the model is assigned to, then the host CPU
+fit.devices[0] // { id, name, unified, usedBytes, freeBytes, totalBytes }
+fit.stats // { ctxSize, gpuLayers }
 ```
 
-`config` is the load, exactly as `loadModel` takes it. The projection resolves it with the same code that load runs, so every setting reaches the fitter the way it would reach the engine, and a load the engine would refuse is refused here as `status: "error"` with `unsupported-config`. A setting left out takes llama's default.
+Each device row gives the memory the load takes on that device, as `usedBytes`, and the device's free and total memory. `id` is the PCI bus ID when the backend reports one, and the device name otherwise. `unified` is true for memory the host owns: the CPU, an integrated GPU, a GPU that shares the host's memory, Android's OpenCL GPU, and a device that reports no memory of its own, which takes the host's figures. With `load_mode: 'mmap'`, the host row leaves out the mapped weights, which the OS can evict.
+
+On an iOS device every unified row shares the app's own memory limit, `os_proc_available_memory()`. Each such row's free memory is capped at that limit, and a load whose unified rows together take more than the limit less the margin does not fit. A limit of 0 means the app has none left.
+
+`stats` is the load as the fit resolved it. On `fits`, `ctxSize` and `gpuLayers` are the context and GPU layer count the fitter settled on, which can be smaller than the load asked for. On `does-not-fit` they are the ones the load asked for.
+
+`status` is `unknown` for a load the fitter cannot project: one placed on the CPU, one with a LoRA adapter or a multimodal projector, one with `fit` turned off, or one the fitter cannot break down per device, the host CPU included. `devices` is then empty and `stats` is `{}`.
+
+The projection reads the model's vocabulary, and llama asserts on a vocabulary it finds inconsistent. An assert aborts the process instead of rejecting, so take the model from a source you trust, or call `assessFit` behind a process boundary.
+
+`config` is the load, exactly as `loadModel` takes it. The projection resolves it with the same code that load runs, so every setting reaches the fitter the way it would reach the engine, and a load the engine would refuse rejects here. A setting left out takes llama's default.
 
-With `spec-type: 'draft-mtp'` the MTP draft context is fitted next to the model, as the load fits it, and its memory is in the verdict and in the per-device figures; on a model or head without MTP layers the load fails, so the projection is refused with `unsupported-config`. A DFlash draft model or a Gemma 4 assistant is fitted the same way, measured next to the model since it reads the model's tensors, and is in the verdict and the figures too (see [Speculative decoding](#speculative-decoding-mtp-dflash)).
+With `spec-type: 'draft-mtp'` the MTP draft context is fitted next to the model, as the load fits it, and its memory is in the verdict and in the device rows; on a model or head without MTP layers the load fails, so the projection rejects. A DFlash draft model or a Gemma 4 assistant is fitted the same way, measured next to the model since it reads the model's tensors, and is in the verdict and the rows too (see [Speculative decoding](#speculative-decoding-mtp-dflash)).
 
-Pinning `gpu-layers` fixes the placement: the fitter refuses to move layers off a device the load claimed, so the answer is whether that exact placement fits, never a reduced one. Leave it unset for a projection that can place the model itself.
+Pinning `gpu-layers` fixes the placement: the fitter keeps layers on the devices the load claimed, so the answer is for that exact placement. Leave it unset for a projection that can place the model itself.
 
-`minCtxSize` sets a floor the fitter may not reduce the context below, defaulting to llama's own `fit-ctx` of 4096. `marginBytes` is the memory to leave free on every device, defaulting to llama's `fit-target` of 1 GiB per device. `backendsDir` is where the dynamically-loaded ggml backends live.
+`minCtxSize` sets a floor the fitter may not reduce the context below, defaulting to llama's own `fit-ctx` of 4096. `marginBytes` is the memory to leave free on every device. The load's own `fit-target`, 1 GiB per device by default, applies too, and the larger of the two binds. `backendsDir` is where the dynamically-loaded ggml backends live.
 
 `assessFit` returns a Promise. Concurrent calls run one at a time within this addon.
 
-A model the fitter cannot read resolves to `status: "error"`; only a broken request rejects.
+A model the fitter cannot read, an invalid request, or a machine with no backend device rejects with an error whose message starts with `assessFit:` and names the cause.
 
 ## Fine-tuning
 
@@ -703,7 +759,7 @@ npm run test:cpp:models:ci   # exactly what .github/workflows/cpp-tests-llm.yml
                              # downloads; matches what CI exercises
 ```
 
-First-run downloads pull several GB from Hugging Face. Every fixture is SHA256-verified against a digest pinned in `scripts/download-unit-test-models.js`; mismatched or partial files are re-downloaded automatically. Set `HF_TOKEN` if a repo requires authentication. Override paths with env vars such as `SHARDED_MODEL_FIRST_SHARD_PATH` (see `test/unit/test_common.hpp`).
+First-run downloads pull several GB, mostly from Hugging Face. Every fixture is SHA256-verified against a digest pinned in `scripts/download-unit-test-models.js`; mismatched or partial files are re-downloaded automatically. Set `HF_TOKEN` if a repo requires authentication. Fixtures pinned by `s3Path` (currently `bitnet_b1_58-large-TQ2_0.gguf`) are copied from the QVAC model registry bucket and need `MODEL_S3_BUCKET` plus AWS credentials; if either is missing or the copy fails, a local run skips the fixture with a warning and its tests skip, while `--ci` fails. Override paths with env vars such as `SHARDED_MODEL_FIRST_SHARD_PATH` or `BITNET_MODEL_PATH` (see `test/unit/test_common.hpp`).
 
 ### Fuzzing
 

package/addon.d.ts

@@ -14,6 +14,7 @@ export interface AddonMessage {
     generationParams?: GenerationParams;
     cacheKey?: string;
     ephemeral?: boolean;
+    baseCacheKey?: string;
 }
 export interface AddonMediaMessage {
     type: "media";
@@ -65,6 +66,12 @@ export interface AddonConfigurationParams {
     config: Record<string, unknown>;
 }
 export type AddonOutputCallback = (addon: unknown, event: unknown, data: unknown, error: unknown, jobId: unknown) => void;
+/** `calcTokens` input as the native binding takes it: `history` and `tools` are JSON array text. */
+export interface AddonCalcTokensParams {
+    text?: string;
+    history?: string;
+    tools?: string;
+}
 export interface LlamaBinding {
     createInstance(owner: LlamaInterface, configurationParams: AddonConfigurationParams, outputCallback: AddonOutputCallback, reserved: null): object;
     loadWeights(handle: unknown, data: LoadWeightsData): Promise<void> | void;
@@ -76,6 +83,7 @@ export interface LlamaBinding {
     finetune?(handle: unknown, params: FinetuneOptions): Promise<number | false> | number | false;
     saveCache(handle: unknown, cacheKey: string): Promise<void>;
     discardCache(handle: unknown, cacheKey: string): Promise<void>;
+    calcTokens(handle: unknown, params: AddonCalcTokensParams): Promise<number>;
     runJob(handle: unknown, data: AddonRunJobMessage[]): Promise<AddonRunJobResult>;
     runJob(handle: unknown, data: AddonBatchRunItem[]): Promise<AddonBatchRunResult>;
     destroyInstance(handle: unknown): void;
@@ -138,6 +146,10 @@ export declare class LlamaInterface {
      * (see `LlmLlamacpp.discardCache`).
      */
     discardCache(cacheKey: string): Promise<void>;
+    /**
+     * Tokens `params` takes on the loaded model (see `LlmLlamacpp.calcTokens`).
+     */
+    calcTokens(params: AddonCalcTokensParams): Promise<number>;
     /**
      * Run finetuning when native binding provides support.
      */

package/addon.js

@@ -154,6 +154,14 @@ class LlamaInterface {
             return Promise.reject(new Error("Model is not loaded"));
         return Promise.resolve(this._binding.discardCache(this._handle, cacheKey));
     }
+    /**
+     * Tokens `params` takes on the loaded model (see `LlmLlamacpp.calcTokens`).
+     */
+    calcTokens(params) {
+        if (!this._handle)
+            return Promise.reject(new Error("Model is not loaded"));
+        return Promise.resolve(this._binding.calcTokens(this._handle, params));
+    }
     /**
      * Run finetuning when native binding provides support.
      */

package/fit.d.ts

@@ -5,41 +5,42 @@ export interface LlamaFitRequest {
     config?: Record<string, string>;
     /** Floor the fitter may not reduce an unpinned context below. */
     minCtxSize?: number;
-    /** Memory to leave free on every device, held to against the load's own `fit-target`. */
+    /** Memory to leave free on every device. The larger of this and the load's own `fit-target` binds. */
     marginBytes?: number;
 }
-export type LlamaFitStatus = 'fits' | 'does-not-fit' | 'error';
-export type LlamaFitReason = 'fits' | 'does-not-fit' | 'model-unreadable' | 'no-backend-device' | 'unsupported-config';
+/** `unknown` is a load the fitter cannot project, such as one placed on the CPU. */
+export type LlamaFitStatus = 'fits' | 'does-not-fit' | 'unknown';
 export interface LlamaFitDevice {
+    /** The PCI bus ID when the backend reports one, the device name otherwise. */
+    id: string;
     name: string;
-    totalBytes: number;
+    unified: boolean;
+    /** Model, context and compute memory the load takes on this device. */
+    usedBytes: number;
     freeBytes: number;
-    modelBytes: number;
-    contextBytes: number;
-    computeBytes: number;
+    totalBytes: number;
 }
+/**
+ * The load as the fit resolved it: on `fits` the context and GPU layer count the
+ * fitter settled on, on `does-not-fit` the ones the load asked for. Empty when
+ * `unknown`.
+ */
+export type LlamaFitStats = {
+    ctxSize: number;
+    gpuLayers: number;
+} | Record<string, never>;
 export interface LlamaFitResult {
     status: LlamaFitStatus;
-    reason: LlamaFitReason;
-    /** The layer count the placement resolved to. */
-    gpuLayers: number;
-    ctxSize: number;
-    /** One row per device the model was assigned to, then a `host` row. */
+    /** One row per device the model is assigned to, then the host CPU. Empty when `unknown`. */
     devices: LlamaFitDevice[];
-    /** Model, context and compute summed across the devices, host excluded. */
-    deviceBytes: number;
-    /** The same, for the trailing host row. */
-    hostBytes: number;
-    trainCtxSize: number;
-    expertCount: number;
+    stats: LlamaFitStats;
 }
 /**
  * Projects one model against the memory free right now, reading GGUF metadata
  * and never weight data. The registry's weightless copy of a model answers the
  * same as the model itself, so this can run before anything is downloaded.
  *
- * A model the fitter cannot read resolves to `status: "error"`; only a broken
- * request rejects.
+ * A model the fitter cannot read, or an invalid request, rejects.
  *
  * Builds the model twice without allocating weights. Concurrent calls run one
  * at a time.

package/fit.js

@@ -6,8 +6,7 @@ exports.assessFit = assessFit;
  * and never weight data. The registry's weightless copy of a model answers the
  * same as the model itself, so this can run before anything is downloaded.
  *
- * A model the fitter cannot read resolves to `status: "error"`; only a broken
- * request rejects.
+ * A model the fitter cannot read, or an invalid request, rejects.
  *
  * Builds the model twice without allocating weights. Concurrent calls run one
  * at a time.

package/index.d.ts

@@ -1,6 +1,6 @@
 import QvacLogger = require("@qvac/logging");
 import { QvacResponse } from "@qvac/infer-base";
-import { assessFit as assessFitImpl, type LlamaFitDevice, type LlamaFitRequest, type LlamaFitResult, type LlamaFitStatus } from "./fit";
+import { assessFit as assessFitImpl, type LlamaFitDevice, type LlamaFitRequest, type LlamaFitResult, type LlamaFitStats, type LlamaFitStatus } from "./fit";
 import type * as AddonModule from "./addon";
 type BareEventMap = Record<string | symbol, unknown[]>;
 /** Aliases: inside the namespace, `QvacResponse` resolves to its own member. */
@@ -9,6 +9,7 @@ type InferQvacResponseOf<Output> = QvacResponse<Output>;
 /** Aliases so the class expression's body can name the namespace types below. */
 type BatchPrompt = LlmLlamacpp.BatchPrompt;
 type BatchResponse = LlmLlamacpp.BatchResponse;
+type CalcTokensInput = LlmLlamacpp.CalcTokensInput;
 type FinetuneHandle = LlmLlamacpp.FinetuneHandle;
 type FinetuneOptions = LlmLlamacpp.FinetuneOptions;
 type LlmLlamacppArgs = LlmLlamacpp.LlmLlamacppArgs;
@@ -70,6 +71,21 @@ interface LlmLlamacpp {
      * remove the conversation from disk.
      */
     discardCache(cacheKey: string): Promise<void>;
+    /**
+     * Tokens `input` takes on the loaded model, without running it. `text`
+     * alone is counted raw, with no chat template and no BOS. With `history` or
+     * `tools` the conversation, `text` (even empty) appended as the last user
+     * turn, is rendered by the chat template as the opening request of a fresh
+     * conversation, with BOS and the generation prompt, as `run()` renders it.
+     * Tools the template leaves out are not counted. `tools` alone has no
+     * message to render and rejects, as `run()` does. Never queues behind
+     * `run()`: it reads only load-time state. A wrong-typed argument rejects
+     * with `TypeError`, and before `load()` it rejects with `Model is not
+     * loaded`. Other invalid input, a media message included, rejects with code
+     * `InvalidArgument`; a conversation the chat template cannot render fails
+     * as it would in `run()`.
+     */
+    calcTokens(input: CalcTokensInput): Promise<number>;
     getState(): {
         configLoaded: boolean;
     };
@@ -364,6 +380,19 @@ declare namespace LlmLlamacpp {
          * maximum is 1048576 (1 TiB). Also accepted as `cache-ram-mib`.
          */
         cache_ram_mib?: NumericLike;
+        /**
+         * Whether a request on a decoder-only model may start from state another
+         * conversation left: the memory a request without a `cacheKey` keeps,
+         * another key's state in a sequence or the RAM tier, or a `cacheKey` file
+         * the model has used. `'true'` (default) shares it, as llama-server's
+         * prompt cache does. `'false'` keeps every request to its own key's state,
+         * for a model serving callers who must not learn through `cacheTokens` or
+         * timing whether another conversation shares their prompt: a request
+         * without a key starts cold and is reset afterwards, a new key starts cold,
+         * and `baseCacheKey` has no effect. Also accepted as
+         * `cache-share-across-keys`.
+         */
+        cache_share_across_keys?: "true" | "false" | "on" | "off" | "1" | "0";
         /**
          * Speculative decoding, as llama-server's `--spec-type`. `'draft-mtp'`
          * drafts with the model's own multi-token-prediction (MTP / NextN) head,
@@ -421,9 +450,30 @@ declare namespace LlmLlamacpp {
             rejectWhenBusy?: boolean;
         };
     }
+    /**
+     * One ordered piece of a message's `content`. Media parts keep their place in
+     * the turn: a single user message can interleave text and media, which a
+     * string `content` plus separate `type: "media"` messages cannot express
+     * without splitting the turn.
+     */
+    type UserContentPart = {
+        type: "text";
+        text: string;
+    } | {
+        type: "media";
+        /**
+         * Either an absolute path to the media file (`string`) or its raw bytes
+         * (`Uint8Array`), the same two forms a `type: "media"` message accepts.
+         */
+        content: string | Uint8Array;
+    };
     interface UserTextMessage {
         role: "system" | "assistant" | "user" | "tool" | "session" | string;
-        content: string;
+        /**
+         * Plain text, or ordered text/media parts rendered as one turn. Media parts
+         * are only valid on user messages.
+         */
+        content: string | UserContentPart[];
         /**
          * An assistant turn's reasoning, separate from its answer in `content`, as
          * in OpenAI-compatible APIs. Optional: an answer sent back with its
@@ -467,6 +517,15 @@ declare namespace LlmLlamacpp {
         parameters?: Record<string, any>;
     }
     type Message = UserTextMessage | UserMediaMessage | ChatFunctionDefinition;
+    /** What `calcTokens` counts. At least one field is required; `tools` needs `text` or `history`. */
+    interface CalcTokensInput {
+        /** Counted raw on its own; with `history` or `tools`, the last user turn. */
+        text?: string;
+        /** A conversation in `run()`'s `Message[]` shape, tool definitions allowed. */
+        history?: Message[];
+        /** Tool definitions, as `run()` takes them. A name also in `history` rejects. */
+        tools?: ChatFunctionDefinition[];
+    }
     interface GenerationParams {
         temp?: number;
         top_p?: number;
@@ -529,10 +588,9 @@ declare namespace LlmLlamacpp {
          * Run prefill only (cache warming): the prompt is evaluated but no tokens
          * are generated. On a model loaded with `parallel >= 2` a prefill needs a
          * `cacheKey` (the warmed conversation stays in its slot for the next
-         * request on that key) — a keyless prefill warms context state that no
-         * concurrent job could reach and is rejected with `InvalidArgument`; run
-         * keyless prefills on a `parallel: 1` model. The same rule applies per
-         * batch item.
+         * request on that key) — a keyless prefill warms state no later request
+         * can count on and is rejected with `InvalidArgument`; run keyless
+         * prefills on a `parallel: 1` model. The same rule applies per batch item.
          */
         prefill?: boolean;
         generationParams?: GenerationParams;
@@ -549,11 +607,26 @@ declare namespace LlmLlamacpp {
          * otherwise be lost with turns the file lacks: when it is set aside for
          * another key, evicted from its batch slot or from the RAM tier, or the
          * model is reloaded or unloaded. With `ephemeral: true` each of those
-         * drops it instead, so its next request starts cold. It still loads an
-         * existing file, and an explicit `saveCache(cacheKey)` still writes it.
+         * drops it instead, so its next request finds no state of its own (it
+         * still reuses any prefix it shares with what the model holds, except on
+         * encoder-decoder models). It still loads an existing file, and an
+         * explicit `saveCache(cacheKey)` still writes it.
          * The flag belongs to the conversation's latest request.
          */
         ephemeral?: boolean;
+        /**
+         * A `cacheKey` file to start from when nothing in memory serves more of
+         * the prompt, such as a system prompt and tools primed with `prefill`.
+         * It is read, never written: the request's own state goes to its
+         * `cacheKey`. A file that is missing or shares no prefix with the prompt
+         * is skipped. Once named, the file stays a source for later requests on
+         * this model, as any `cacheKey` file it used does, until files named
+         * or picked as a prompt's closest match more recently push it out (the
+         * model keeps the 256 most recent, as far as 64 MiB of memory holds
+         * their prompt records). No effect with `cache_share_across_keys:
+         * 'false'`.
+         */
+        baseCacheKey?: string;
         /**
          * Admission policy when the worker pool is full. `true` rejects before
          * submitting with an `Error` carrying `code === 'RUN_BUSY'` — branch on the
@@ -763,5 +836,6 @@ declare namespace LlmLlamacpp {
     type FitResult = LlamaFitResult;
     type FitStatus = LlamaFitStatus;
     type FitDevice = LlamaFitDevice;
+    type FitStats = LlamaFitStats;
 }
 export = LlmLlamacpp;

package/index.js

@@ -16,6 +16,7 @@ function normalizeRunOptions(runOptions) {
             generationParams: undefined,
             cacheKey: undefined,
             ephemeral: false,
+            baseCacheKey: undefined,
             rejectWhenBusy: undefined,
         };
     }
@@ -43,6 +44,9 @@ function normalizeRunOptions(runOptions) {
     if (options.ephemeral !== undefined && typeof options.ephemeral !== "boolean") {
         throw new TypeError("ephemeral must be a boolean when provided");
     }
+    if (options.baseCacheKey !== undefined && typeof options.baseCacheKey !== "string") {
+        throw new TypeError("baseCacheKey must be a string when provided");
+    }
     if (options.rejectWhenBusy !== undefined && typeof options.rejectWhenBusy !== "boolean") {
         throw new TypeError("rejectWhenBusy must be a boolean when provided");
     }
@@ -51,15 +55,28 @@ function normalizeRunOptions(runOptions) {
         generationParams: normalizeGenerationParams(options.generationParams),
         cacheKey: options.cacheKey,
         ephemeral: options.ephemeral === true,
+        baseCacheKey: options.baseCacheKey,
         // Left undefined when unset so admission falls back to the instance default.
         rejectWhenBusy: options.rejectWhenBusy,
     };
 }
+// Malformed parts (null, non-objects) are not byte media: they pass through
+// untouched so the native validator rejects them with `InvalidInputFormat`.
+function isByteMediaPart(part) {
+    const candidate = part;
+    return (candidate !== null &&
+        typeof candidate === "object" &&
+        candidate.type === "media" &&
+        candidate.content instanceof Uint8Array);
+}
 function promptToAddonMessages(prompt, runOptions) {
     if (!Array.isArray(prompt)) {
         throw new TypeError("Prompt input must be Message[]");
     }
-    const { prefill, generationParams, cacheKey, ephemeral } = normalizeRunOptions(runOptions);
+    const { prefill, generationParams, cacheKey, ephemeral, baseCacheKey } = normalizeRunOptions(runOptions);
+    // Only ever JSON-serialized into the addon's text message: a byte-buffer
+    // media part reaches the native side path-less, bound to the hoisted buffer
+    // by position, so the wire shape is wider than the public `Message` type.
     const textMessages = [];
     const mediaItems = [];
     for (const message of prompt) {
@@ -67,10 +84,26 @@ function promptToAddonMessages(prompt, runOptions) {
         if (media.role === "user" && media.type === "media" && media.content instanceof Uint8Array) {
             mediaItems.push(media.content);
             textMessages.push({ ...media, content: "" });
+            continue;
         }
-        else {
-            textMessages.push(message);
+        // Ordered parts: hoist each media part's bytes next to the other media
+        // messages and leave a path-less part behind. The native side records the
+        // plan in marker order, so the bytes still bind to the right markers.
+        const text = message;
+        if (Array.isArray(text.content)) {
+            textMessages.push({
+                ...text,
+                content: text.content.map((part) => {
+                    if (isByteMediaPart(part)) {
+                        mediaItems.push(part.content);
+                        return { type: "media" };
+                    }
+                    return part;
+                }),
+            });
+            continue;
         }
+        textMessages.push(message);
     }
     const promptMessages = [];
     for (const mediaData of mediaItems) {
@@ -83,6 +116,7 @@ function promptToAddonMessages(prompt, runOptions) {
         generationParams,
         cacheKey,
         ephemeral,
+        baseCacheKey,
     });
     return promptMessages;
 }
@@ -257,6 +291,15 @@ function sanitizePromptForLog(prompt) {
         if (media && media.type === "media" && media.content instanceof Uint8Array) {
             return { ...media, content: `[Uint8Array byteLength=${media.content.byteLength}]` };
         }
+        const text = msg;
+        if (Array.isArray(text?.content)) {
+            return {
+                ...text,
+                content: text.content.map((part) => isByteMediaPart(part)
+                    ? { ...part, content: `[Uint8Array byteLength=${part.content.byteLength}]` }
+                    : part),
+            };
+        }
         return msg;
     });
 }
@@ -826,6 +869,62 @@ const LlmLlamacpp = class LlmLlamacpp {
         }
         await this.addon.discardCache(cacheKey);
     }
+    /**
+     * Tokens `input` takes on the loaded model, without running it. `text`
+     * alone is counted raw, with no chat template and no BOS. With `history` or
+     * `tools` the conversation, `text` (even empty) appended as the last user
+     * turn, is rendered by the chat template as the opening request of a fresh
+     * conversation, with BOS and the generation prompt, as `run()` renders it.
+     * Tools the template leaves out are not counted. `tools` alone has no
+     * message to render and rejects, as `run()` does. Never queues behind
+     * `run()`: it reads only load-time state. A wrong-typed argument rejects
+     * with `TypeError`, and before `load()` it rejects with `Model is not
+     * loaded`. Other invalid input, a media message included, rejects with code
+     * `InvalidArgument`; a conversation the chat template cannot render fails
+     * as it would in `run()`.
+     */
+    async calcTokens(input) {
+        if (input === null || typeof input !== "object") {
+            throw new TypeError("calcTokens(input) requires an object");
+        }
+        const text = input.text ?? undefined;
+        const history = input.history ?? undefined;
+        const tools = input.tools ?? undefined;
+        if (text !== undefined && typeof text !== "string") {
+            throw new TypeError("calcTokens: text must be a string");
+        }
+        if (history !== undefined && !Array.isArray(history)) {
+            throw new TypeError("calcTokens: history must be a Message[]");
+        }
+        if (tools !== undefined && !Array.isArray(tools)) {
+            throw new TypeError("calcTokens: tools must be an array of function definitions");
+        }
+        if (!this.addon) {
+            throw new Error("Model is not loaded");
+        }
+        const params = {};
+        if (text !== undefined)
+            params.text = text;
+        // Media bytes are dropped rather than serialized: the native side rejects
+        // any media message or ordered media part by its type.
+        if (history !== undefined) {
+            params.history = JSON.stringify(history.map((message) => {
+                if (message?.type === "media")
+                    return { ...message, content: "" };
+                const textMessage = message;
+                if (Array.isArray(textMessage?.content)) {
+                    return {
+                        ...message,
+                        content: textMessage.content.map((part) => isByteMediaPart(part) ? { type: "media" } : part),
+                    };
+                }
+                return message;
+            }));
+        }
+        if (tools !== undefined)
+            params.tools = JSON.stringify(tools);
+        return this.addon.calcTokens(params);
+    }
     getState() {
         return this.state;
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@qvac/llm-llamacpp",
-  "version": "0.65.1",
+  "version": "0.67.0",
   "description": "llama addon for qvac",
   "addon": true,
   "scripts": {