@qvac/llm-llamacpp 0.56.0 → 0.58.0

package/README.md

@@ -32,11 +32,26 @@ This native C++ addon, built using the `Bare` Runtime, simplifies running Large
 |----------|-------------|-------------|--------|-------------|
 | macOS | arm64, x64 | 14.0+ | ✅ Tier 1 | Metal |
 | iOS | arm64 | 17.0+ | ✅ Tier 1 | Metal |
-| Linux | arm64, x64 | Ubuntu-22+ | ✅ Tier 1 | Vulkan |
+| Linux | arm64, x64 | Ubuntu-22+ | Tier 1 | CUDA (NVIDIA), Vulkan |
 | Android | arm64 | 12+ | ✅ Tier 1 | Vulkan, OpenCL (Adreno 700+) |
-| Windows | x64 | 10+ | ✅ Tier 1 | Vulkan |
+| Windows | x64 | 10+ | Tier 1 | CUDA (NVIDIA), Vulkan |
 
 
+**Note on CUDA (Linux and Windows, NVIDIA):**
+The CUDA backend ships as a dynamically loaded module alongside Vulkan, and is preferred over
+Vulkan when an NVIDIA device is present. On Windows it needs a CUDA 13 install: the runtime DLLs
+are loaded from `%CUDA_PATH%\bin\x64`, not from `PATH`.
+
+- If the CUDA module, driver, or required runtime DLL is missing, the device never registers and selection
+  falls through to Vulkan, then CPU. Nothing needs configuring for that.
+- `backend: "vulkan"` forces Vulkan on an NVIDIA machine. Setting `CUDA_VISIBLE_DEVICES=-1` in the
+  environment has the same effect without touching the load config.
+- TurboQuant / PolarQuant KV-cache types (`tbq3_0`, `tbq4_0`, `pq3_0`, `pq4_0`) are **not**
+  supported on CUDA and are rejected during model configuration. Use a Vulkan GPU or CPU for those.
+  Standard quantized types (`q4_0`, `q8_0`, …) work normally.
+- BitNet (TQ1_0 / TQ2_0) and some LoRA finetuning kernels are not yet available on CUDA. Use
+  `backend: "vulkan"` for those workloads until the kernels land.
+
 **Note — BitNet models (TQ1_0 / TQ2_0 quantization):**
 BitNet models require special backend handling on Adreno GPUs. When a BitNet model is detected and no explicit `main-gpu` is set:
 - **Adreno 800+** (e.g. Adreno 830): Vulkan is used instead of OpenCL.
@@ -177,9 +192,15 @@ const config = {
 | tools             | `"true"` or `"false"`                       | `"false"`                    | Enable tool calling with jinja templating             |
 | verbosity         | 0 – 3 (0=ERROR, 1=WARNING, 2=INFO, 3=DEBUG) | 0                            | Logging verbosity level                               |
 | main-gpu          | integer, `"integrated"`, or `"dedicated"`   | —                            | GPU selection for multi-GPU systems                   |
+| backend           | comma-separated list of `cuda`, `vulkan`, `metal`, `opencl`, or `auto` | N/A | Overrides which GPU backend is used, in priority order (e.g. `"cuda,vulkan"`). `auto` means no preference. An unrecognised name is rejected; a recognised one with no device present is skipped. Use `device: "cpu"` to run on CPU. Cannot be combined with `split-mode` or `devices` |
 | split-mode        | `"none"`, `"layer"`, or `"tensor"` | `"none"`                     | How to split the model across GPUs. `"tensor"` is EXPERIMENTAL and desktop-only; `"row"` is rejected ([details](./docs/multi-gpu.md)) |
 | tensor-split      | comma-separated proportions (e.g. `"1,1"`)  | —                            | GPU split ratios for the multi-GPU split modes ([details](./docs/multi-gpu.md)) |
 | parallel          | integer                                     | 1                            | Concurrent sequence slots for continuous batching. Values `>= 2` enable batch `run()` and split the KV cache uniformly across slots ([details](./docs/continuous-batching.md)) |
+| cache_checkpoints | 0 – 1024                                    | 1                            | Per-sequence cap on process-local checkpoints kept for cached requests on hybrid / recurrent models and DeepSeek V4, one per committed request at the end of its history. Each holds only the state a tail trim cannot rebuild (about 20 MB on Qwen3.5-0.8B). `1` serves the next turn and a regenerate; editing the user message k-th from the end (1 = the last) without reprocessing the whole conversation needs `k + 1`, so `2` for the last one; `0` disables them. Also accepted as `cache-checkpoints` ([details](./docs/cache-api.md), [how it works](./docs/cache-lifecycle.md)) |
+| cache_checkpoints_max_bytes | integer (bytes)                 | 0 (unlimited)                | Byte budget for those checkpoints, enforced before the count. The load fails with `InvalidArgument` when it cannot hold `cache_checkpoints` checkpoints of the largest size the context allows. Also accepted as `cache-checkpoints-max-bytes` |
+| cache_checkpoint_storage | `"memory"` or `"disk"`             | `"memory"`                   | Where checkpoints and the per-request rollback snapshot live. `"memory"` keeps them in host RAM so a cached chat never touches the disk; `"disk"` writes them to a private directory inside `cache_checkpoint_dir`, which it requires. Also accepted as `cache-checkpoint-storage` |
+| cache_checkpoint_dir | directory path                              | none                         | **Required with `cache_checkpoint_storage: "disk"`**, and refused without it: where checkpoint files go, inside a private directory the addon creates in it (mode 0700; on Windows, the directory itself). On mobile, pass the app's cache directory. The load fails with `InvalidArgument` when it is missing, empty, or unusable. Also accepted as `cache-checkpoint-dir` |
+| cache_ram_mib | 0 – 1048576 (MiB)                               | 0 (off)                      | Host-RAM budget for conversations that are not running. A `cacheKey` switch or a batch slot eviction moves the conversation there instead of writing its file, and the next request on that key restores it. A write-back cache: unsaved turns reach the file on `saveCache()`, when the budget pushes the entry out, or at unload; ephemeral entries are dropped instead. Also accepted as `cache-ram-mib` ([details](./docs/cache-api.md#keep-switched-away-conversations-in-ram)) |
 | flash-attn        | `"on"`/`"enabled"`/`"true"`/`"1"`, `"off"`/`"disabled"`/`"false"`/`"0"`, or `"auto"` | `"on"`, except when finetuning or on a BitNet model, where it is forced off | Flash attention. The four truthy and four falsey spellings are equivalent; `"auto"` is a third state that defers to qvac-fabric's runtime capability probe. Lower-case only — matching is case-sensitive and any other value is rejected. Affects the KV-cache auto-default — see below. Also accepted as `flash_attn`; supplying both spellings is an error |
 | cache-type-k      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **key** quantization type. Unset = auto-default (see KV-cache type below) |
 | cache-type-v      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **value** quantization type. Quantizing V requires `flash-attn` on |
@@ -192,7 +213,7 @@ The addon picks a safe KV-cache type when `cache-type-k`/`cache-type-v` are unse
 
 - **Auto-default:** on a **Metal / Vulkan GPU** (with flash attention on) both K and V default to **`q8_0`** — quality-neutral vs `f16` and ~47% smaller KV cache. **CPU** and **OpenCL (Adreno)** keep **`f16`** (ARM CPU `q8_0` has a quality/throughput cost; quantized KV is unsafe on OpenCL — see below). Finetuning manages its own KV types and is left untouched.
 - **`flash-attn: 'auto'` keeps `f16`.** "Flash attention on" above means a truthy `flash-attn` — `on`, `enabled`, `true` or `1`, or the `on` default when the key is unset. `'auto'` is deliberately excluded: quantizing the V cache forces qvac-fabric to promote AUTO to ENABLED, which skips the runtime capability probe that `'auto'` exists to run. So `'auto'` trades the ~47% KV-cache saving for letting qvac-fabric decide. To get both, set `cache-type-k`/`-v` explicitly alongside `'auto'`, or use `'on'`. **`split-mode: 'tensor'` is the exception:** qvac-fabric promotes AUTO to ENABLED unconditionally for that mode, so there is no probe to preserve and `'auto'` takes the q8_0 default there exactly as `'on'` does.
-- **OpenCL (Adreno) accepts only `f16`/`f32`/`bf16`:** any other cache type — quantized (`q8_0`, `q4_0`, `q4_1`, `q5_0`, …) or unrecognized — throws a `StatusError`. A quantized K or V cache aborts in `llama_kv_cache::update` on cache management (reasoning-block compaction, state restore) because ggml-opencl has no `F32→quantized` requantize kernel. Use `f16`/`f32`/`bf16`, or a Vulkan GPU / CPU.
+- **OpenCL (Adreno) accepts only `f16`/`f32`/`bf16`:** any other cache type — quantized (`q8_0`, `q4_0`, `q4_1`, `q5_0`, …) or unrecognized — throws a `StatusError`. A quantized K or V cache aborts in `llama_kv_cache::update` during state restore because ggml-opencl has no `F32→quantized` requantize kernel. Use `f16`/`f32`/`bf16`, or a Vulkan GPU / CPU.
 - **Mixed K≠V is a warning, not an error:** if K and V differ and at least one is quantized, the addon logs a warning (asymmetric quantized K/V falls off the fused flash-attention path — a notable GPU decode penalty — for no quality benefit, and is unsupported on Adreno OpenCL) but proceeds. Prefer a symmetric type. (This may be relaxed once qvac-fabric handles asymmetric quantized K/V efficiently.)
 
 
@@ -347,7 +368,7 @@ Admission is controlled by `rejectWhenBusy` (instance-level `opts.rejectWhenBusy
 
 #### Prefill (cache warming) with `parallel >= 2`
 
-A prefill-only run (`runOptions.prefill: true`) is admitted on a parallel model only when its product survives the slot teardown, i.e. it is *persistable*: `saveCacheToDisk: true` plus a `cacheKey`. A live-only prefill (no persistence) warms context state that no concurrent job could ever reach, so it is rejected with `InvalidArgument`; run it on a `parallel: 1` model instead. The same rule applies per item in batch runs. See [cache-api.md](./docs/cache-api.md).
+A prefill-only run (`runOptions.prefill: true`) is admitted on a parallel model only when its product survives the slot teardown, i.e. it has a `cacheKey`: the warmed conversation stays in its slot for the next request on that key. A keyless prefill warms context state that no concurrent job could ever reach, so it is rejected with `InvalidArgument`; run it on a `parallel: 1` model instead. The same rule applies per item in batch runs. See [cache-api.md](./docs/cache-api.md).
 
 #### Cancelling a batch
 

package/addon.d.ts

@@ -13,7 +13,7 @@ export interface AddonMessage {
      */
     generationParams?: GenerationParams;
     cacheKey?: string;
-    saveCacheToDisk?: boolean;
+    ephemeral?: boolean;
 }
 export interface AddonMediaMessage {
     type: "media";
@@ -74,6 +74,8 @@ export interface LlamaBinding {
     cancel(handle: unknown, savePauseCheckpoint: number): Promise<void> | void;
     cancelJob(handle: unknown, id: number): Promise<void> | void;
     finetune?(handle: unknown, params: FinetuneOptions): Promise<number | false> | number | false;
+    saveCache(handle: unknown, cacheKey: string): Promise<void>;
+    discardCache(handle: unknown, cacheKey: string): Promise<void>;
     runJob(handle: unknown, data: AddonRunJobMessage[]): Promise<AddonRunJobResult>;
     runJob(handle: unknown, data: AddonBatchRunItem[]): Promise<AddonBatchRunResult>;
     destroyInstance(handle: unknown): void;
@@ -127,6 +129,16 @@ export declare class LlamaInterface {
      * jobs running. Routes to MultiJobScheduler::cancel(id) -> cancelById(id).
      */
     cancelJob(id: number): Promise<void>;
+    /**
+     * Write the conversation kept in memory for `cacheKey` to its file (see
+     * `LlmLlamacpp.saveCache`).
+     */
+    saveCache(cacheKey: string): Promise<void>;
+    /**
+     * Drop the conversation kept in memory for `cacheKey` without writing it
+     * (see `LlmLlamacpp.discardCache`).
+     */
+    discardCache(cacheKey: string): Promise<void>;
     /**
      * Run finetuning when native binding provides support.
      */

package/addon.js

@@ -160,6 +160,24 @@ class LlamaInterface {
         }
         await this._binding.cancelJob(this._handle, id);
     }
+    /**
+     * Write the conversation kept in memory for `cacheKey` to its file (see
+     * `LlmLlamacpp.saveCache`).
+     */
+    saveCache(cacheKey) {
+        if (!this._handle)
+            return Promise.reject(new Error("Model is not loaded"));
+        return Promise.resolve(this._binding.saveCache(this._handle, cacheKey));
+    }
+    /**
+     * Drop the conversation kept in memory for `cacheKey` without writing it
+     * (see `LlmLlamacpp.discardCache`).
+     */
+    discardCache(cacheKey) {
+        if (!this._handle)
+            return Promise.reject(new Error("Model is not loaded"));
+        return Promise.resolve(this._binding.discardCache(this._handle, cacheKey));
+    }
     /**
      * Run finetuning when native binding provides support.
      */

package/index.d.ts

@@ -55,6 +55,21 @@ interface LlmLlamacpp {
     cancel(): Promise<void>;
     pause(): Promise<void>;
     unload(): Promise<void>;
+    /**
+     * Write the conversation kept in memory for `cacheKey` to that file now.
+     * Waits for a request running on that key, not one still queued, so read a
+     * response to the end first. Writes only when the file lacks turns, and
+     * rejects when nothing is cached under the key or the write fails (the
+     * conversation then stays in memory, still unsaved).
+     */
+    saveCache(cacheKey: string): Promise<void>;
+    /**
+     * Drop the conversation kept in memory for `cacheKey` without writing it,
+     * so no later eviction or unload writes it. Waits for a request running on
+     * that key. The file, if one was written, is left alone: delete it too to
+     * remove the conversation from disk.
+     */
+    discardCache(cacheKey: string): Promise<void>;
     getState(): {
         configLoaded: boolean;
     };
@@ -117,6 +132,13 @@ declare namespace LlmLlamacpp {
     }
     interface LlamaConfig {
         device?: string;
+        /**
+         * Comma-separated GPU backend priority list, e.g. `'cuda,vulkan'`.
+         * Accepted names: cuda, vulkan, metal, opencl, plus
+         * `auto` for no preference. An unrecognised name is rejected; a recognised
+         * one with no device present is skipped. Use `device: 'cpu'` to run on CPU.
+         */
+        backend?: string;
         gpu_layers?: NumericLike;
         ctx_size?: NumericLike;
         system_prompt?: string;
@@ -264,6 +286,63 @@ declare namespace LlmLlamacpp {
          * `InvalidArgument` naming both knobs.
          */
         parallel?: NumericLike;
+        /**
+         * Per-sequence cap on the process-local full-state checkpoints kept for
+         * cached requests on hybrid / recurrent models (Qwen3.5, Jamba,
+         * Granite-Hybrid, DeepSeek V4, ...). Such models cannot trim a KV tail,
+         * so a diverging history is served by restoring the longest checkpoint
+         * that is still a prefix of the new prompt. A committed cached request
+         * adds one, at the end of its history, holding only the state a tail
+         * trim cannot rebuild. The default is 1: the last one, which an ordinary
+         * next turn and a regenerate restore. `2` also keeps the one before it,
+         * which an edit of the last user message restores. Changing the user
+         * message k-th from the end (1 = the last) without reprocessing the whole
+         * conversation needs at least k + 1; with the default 1, editing the last
+         * user message reprocesses everything. `0` keeps and takes none (every
+         * divergent turn is a cold prefill), the maximum is 1024.
+         * Ignored on pure-attention models, which never take checkpoints.
+         * Also accepted as `cache-checkpoints`; supplying both is an error.
+         */
+        cache_checkpoints?: NumericLike;
+        /**
+         * Byte budget for the checkpoints kept per sequence, enforced before
+         * `cache_checkpoints`: the oldest checkpoints are dropped until the total
+         * payload fits. `0` (default) is unlimited. When set, the model load fails
+         * with `InvalidArgument` if the budget cannot hold `cache_checkpoints`
+         * checkpoints of the largest size the context allows, so a misconfigured
+         * budget is reported up front rather than discovered mid-conversation.
+         * Also accepted as `cache-checkpoints-max-bytes`.
+         */
+        cache_checkpoints_max_bytes?: NumericLike;
+        /**
+         * Where checkpoints and the per-request rollback snapshot live:
+         * `'memory'` (default) keeps them in host RAM so a cached chat never
+         * touches the disk, `'disk'` writes them to files in
+         * `cache_checkpoint_dir`, which it requires. Each snapshot holds only the
+         * state a tail trim cannot rebuild, a size fixed by the model (about 20 MB
+         * on Qwen3.5-0.8B). Also accepted as `cache-checkpoint-storage`.
+         */
+        cache_checkpoint_storage?: "disk" | "memory";
+        /**
+         * Required with `cache_checkpoint_storage: 'disk'`, and refused without
+         * it: where checkpoint files go, inside a private directory the addon
+         * creates in it (mode 0700; on Windows, the directory itself). There is
+         * no default, because the OS temp directory is not usable on every
+         * platform (Android); on mobile, pass the app's cache directory. The load
+         * fails with `InvalidArgument` when it is missing, empty, or unusable.
+         * Also accepted as `cache-checkpoint-dir`.
+         */
+        cache_checkpoint_dir?: string;
+        /**
+         * Host-RAM budget, in MiB, for conversations that are not running: a
+         * `cacheKey` switch (single prompt) or a slot eviction (batch) moves the
+         * conversation there instead of writing its file, and the next request on
+         * that key restores it from there. A write-back cache: unsaved turns reach
+         * the file on `saveCache()`, when the budget pushes the entry out, or at
+         * unload; ephemeral entries are dropped instead. `0` (default) is off; the
+         * maximum is 1048576 (1 TiB). Also accepted as `cache-ram-mib`.
+         */
+        cache_ram_mib?: NumericLike;
         [key: string]: string | number | boolean | string[] | undefined;
     }
     interface LlmLlamacppArgs {
@@ -287,6 +366,14 @@ declare namespace LlmLlamacpp {
     interface UserTextMessage {
         role: "system" | "assistant" | "user" | "tool" | "session" | string;
         content: string;
+        /**
+         * An assistant turn's reasoning, separate from its answer in `content`, as
+         * in OpenAI-compatible APIs. Optional: an answer sent back with its
+         * reasoning inline (`<think>…</think>` first, as the model streamed it) is
+         * split into the two, using the reasoning markers the chat template
+         * reports, so the template can drop or place it. Ignored on other roles.
+         */
+        reasoning_content?: string;
         type?: undefined;
         [key: string]: any;
     }
@@ -349,6 +436,12 @@ declare namespace LlmLlamacpp {
          * `"required"` or a function name without tools throws.
          */
         tool_choice?: "auto" | "none" | "required" | (string & {});
+        /**
+         * Whether one response may carry more than one tool call. `true` lets the
+         * template and the tool-call grammar accept several; unset or `false`
+         * keeps one. Ignored when the prompt carries no tools.
+         */
+        parallel_tool_calls?: boolean;
         /**
          * Per-request reasoning channel budget. `-1` keeps the model's reasoning
          * channel on; `0` disables it for this request; any positive integer caps
@@ -357,120 +450,37 @@ declare namespace LlmLlamacpp {
          * value is restored afterwards.
          */
         reasoning_budget?: number;
-        /**
-         * When the model emits a reasoning block during generation (e.g.
-         * `<think>...</think>` for the Qwen3 family, `<|channel>thought ...
-         * <channel|>` for Gemma 4), drop those tokens from the KV cache at
-         * end-of-generation so subsequent turns do not accumulate reasoning
-         * history.
-         *
-         * Defaults to `false` for all models except the Qwen3 reasoning family
-         * (Qwen3, Qwen3.5, and Qwen3.6, including MoE variants), which defaults
-         * to `true`. Set this per-request `generationParams` value to override the
-         * model default. Set to `false` to preserve reasoning tokens in the KV / SSM
-         * cache across turns (e.g. chain-of-thought agents that want the next turn
-         * to attend to prior reasoning, interpretability tooling, or cache-reuse
-         * patterns that depend on the reasoning-inclusive state). Supported on both
-         * text and multimodal contexts. No-op for models without a recognised
-         * reasoning channel.
-         *
-         * Every model kind is handled the same way: the sequence is rewound to a
-         * boundary anchored BEFORE the reasoning span, and the tokens that sit
-         * outside the span, the pre-reasoning preamble and the answer tail, are
-         * replayed through the decoder. Only the anchor's form differs. Recurrent
-         * / hybrid-SSM models (Qwen3.5, Qwen3-Next, Jamba, Granite-Hybrid, ...)
-         * anchor a full-state snapshot, because the recurrent half cannot be
-         * rewound by dropping cells; pure-attention models anchor a bare position
-         * and rewind with a tail trim.
-         *
-         * No structural reasoning marker is seeded or replayed, so the compacted
-         * cache is preamble plus answer with no `<think>` / `</think>` scaffold
-         * left behind, and close-marker length decides nothing: a marker that
-         * tokenises to several pieces is supported like any other. Chat templates
-         * that force-open the reasoning channel during prefill and templates that
-         * let the model generate the opener are both supported; on the
-         * generated-opener path the sampled pieces that open the block are clipped
-         * out of the replay rather than rebuilt.
-         *
-         * Prefill-only (cache-warm) requests anchor nothing: they never enter
-         * generation and cannot emit reasoning tokens.
-         *
-         * Uniform hard-fail contract: any inability to remove the reasoning
-         * span from cache, whether the boundary anchor, the rewind, or the
-         * replay step, is surfaced to the caller as a `StatusError`. There is no
-         * soft-failure counter: if the feature is
-         * enabled and cache cleanup cannot complete, the final request result is
-         * failed rather than reported as a successful answer with the reasoning span
-         * still resident in cache.
-         *
-         * Streaming caveat: token callbacks (`outputCallback` / batch `onToken`) are
-         * invoked during generation, while reasoning-block compaction runs at
-         * end-of-generation. If compaction fails, streaming callers may already have
-         * received partial or complete text. Treat streamed text as tentative until
-         * the request completes successfully; non-streaming callers receive no
-         * successful returned answer on this failure path.
-         *
-         * Before throwing, the affected sequence is cleaned up so that the
-         * next request on the same context starts from a coherent state:
-         *   * Boundary-anchor failure: nothing has been rewound yet, so the
-         *     driver rolls back to its pre-prompt checkpoint (or clears the
-         *     sequence entirely on restore underflow) and resets positional
-         *     accounting, then rethrows.
-         *   * Rewind or replay failure: compaction rewinds before it replays,
-         *     so live KV has already been written to by this point and a tail
-         *     trim can no longer reach a coherent state. The compactor
-         *     best-effort wipes the sequence and the driver zeroes its
-         *     positional accounting to match, so subsequent turns cannot decode
-         *     into contaminated positions.
-         *
-         * On the continuous-batch path, the scheduler's error-recovery leg
-         * deliberately does NOT persist the failed slot's cache: when the
-         * request was configured with `cacheKey` + `saveCacheToDisk`, the
-         * last known-good on-disk cache is preserved rather than being
-         * overwritten with the post-failure state. The same skip-save rule
-         * applies to graceful cancels of hybrid / recurrent requests when
-         * rollback to the pre-request cursor cannot be completed (recurrent
-         * full-state restore refused, or no pre-request snapshot was captured
-         * yet the driver has advanced past the pre-request cursor). Cancels
-         * that can be rolled back cleanly still persist as usual.
-         */
-        remove_thinking_from_context?: boolean;
     }
     interface RunOptions {
         /**
          * Run prefill only (cache warming): the prompt is evaluated but no tokens
-         * are generated. On a model loaded with `parallel >= 2` a prefill is
-         * admitted only when it is *persistable* (`saveCacheToDisk: true` plus a
-         * `cacheKey`) — a live-only prefill warms context state that no concurrent
-         * job could reach and is rejected with `InvalidArgument`; run live-only
-         * prefills on a `parallel: 1` model. The same rule applies per batch item.
+         * are generated. On a model loaded with `parallel >= 2` a prefill needs a
+         * `cacheKey` (the warmed conversation stays in its slot for the next
+         * request on that key) — a keyless prefill warms context state that no
+         * concurrent job could reach and is rejected with `InvalidArgument`; run
+         * keyless prefills on a `parallel: 1` model. The same rule applies per
+         * batch item.
          */
         prefill?: boolean;
         generationParams?: GenerationParams;
+        /**
+         * Enables addon-owned prompt caching at this path. Every cached request
+         * must resend the complete authoritative message history and tool list;
+         * delta-only continuations are not supported. The addon renders once and
+         * decodes only the suffix after the longest matching token/media prefix.
+         */
         cacheKey?: string;
         /**
-         * When `true` and `cacheKey` is set, the driver persists the sequence's
-         * KV / recurrent state to disk under `cacheKey` at end-of-generation so a
-         * later run keyed by the same string can resume without re-prefilling.
-         *
-         * The continuous-batch scheduler intentionally SKIPS the save on
-         * teardown legs where persistence could corrupt the last known-good
-         * on-disk cache:
-         *   - Any batch error-recovery path (e.g. decode failure, per-slot
-         *     failure with `SaveCachePolicy::Skip`, or a
-         *     `remove_thinking_from_context` hard-fail).
-         *   - Graceful cancel of a hybrid / recurrent request whose driver
-         *     cannot roll live memory back to the pre-request cursor —
-         *     either the recurrent full-state restore was refused, or no
-         *     pre-request snapshot exists yet the driver advanced past the
-         *     pre-request cursor. Cancels that roll back cleanly still save.
-         *
-         * On both skip paths the sequence's in-memory KV is still cleared, so
-         * subsequent requests decode from a coherent baseline; only the
-         * on-disk cache is untouched. Pure-attention drivers always roll back
-         * via `removeLastNTokens` and therefore save on cancel as usual.
+         * Keep this conversation in memory only. By default a cached
+         * conversation is written to its `cacheKey` file whenever it would
+         * otherwise be lost with turns the file lacks: when it is set aside for
+         * another key, evicted from its batch slot or from the RAM tier, or the
+         * model is reloaded or unloaded. With `ephemeral: true` each of those
+         * drops it instead, so its next request starts cold. It still loads an
+         * existing file, and an explicit `saveCache(cacheKey)` still writes it.
+         * The flag belongs to the conversation's latest request.
          */
-        saveCacheToDisk?: boolean;
+        ephemeral?: boolean;
         /**
          * Admission policy when the worker pool is full. `true` rejects before
          * submitting with an `Error` carrying `code === 'RUN_BUSY'` — branch on the
@@ -523,15 +533,6 @@ declare namespace LlmLlamacpp {
         CacheTokens: number;
         generatedTokens: number;
         promptTokens: number;
-        /**
-         * Number of `<think>` (or model-equivalent) reasoning blocks dropped
-         * from the KV cache at end-of-generation by the
-         * `remove_thinking_from_context` feature. Per-inference for single
-         * requests; summed across completed slots for batch requests. 0 when
-         * the model has no recognised reasoning channel, when the feature
-         * was disabled per-request, or when no reasoning blocks were emitted.
-         */
-        thinkingBlockDiscards: number;
         /**
          * Number of prompt renders in this request that provably left the tool
          * definitions out — the template either rejected them, or supplying them

package/index.js

@@ -15,7 +15,7 @@ function normalizeRunOptions(runOptions) {
             prefill: false,
             generationParams: undefined,
             cacheKey: undefined,
-            saveCacheToDisk: false,
+            ephemeral: false,
             rejectWhenBusy: undefined,
         };
     }
@@ -35,8 +35,13 @@ function normalizeRunOptions(runOptions) {
     if (options.cacheKey !== undefined && typeof options.cacheKey !== "string") {
         throw new TypeError("cacheKey must be a string when provided");
     }
-    if (options.saveCacheToDisk !== undefined && typeof options.saveCacheToDisk !== "boolean") {
-        throw new TypeError("saveCacheToDisk must be a boolean when provided");
+    if (options.saveCacheToDisk !== undefined) {
+        throw new TypeError("saveCacheToDisk was removed: a cached conversation stays in memory and is written to " +
+            "its cacheKey file when it is set aside, evicted or the model unloads; call " +
+            "saveCache(cacheKey) to write it now, or set ephemeral: true to never write it");
+    }
+    if (options.ephemeral !== undefined && typeof options.ephemeral !== "boolean") {
+        throw new TypeError("ephemeral must be a boolean when provided");
     }
     if (options.rejectWhenBusy !== undefined && typeof options.rejectWhenBusy !== "boolean") {
         throw new TypeError("rejectWhenBusy must be a boolean when provided");
@@ -45,7 +50,7 @@ function normalizeRunOptions(runOptions) {
         prefill: options.prefill === true,
         generationParams: normalizeGenerationParams(options.generationParams),
         cacheKey: options.cacheKey,
-        saveCacheToDisk: options.saveCacheToDisk === true,
+        ephemeral: options.ephemeral === true,
         // Left undefined when unset so admission falls back to the instance default.
         rejectWhenBusy: options.rejectWhenBusy,
     };
@@ -54,7 +59,7 @@ function promptToAddonMessages(prompt, runOptions) {
     if (!Array.isArray(prompt)) {
         throw new TypeError("Prompt input must be Message[]");
     }
-    const { prefill, generationParams, cacheKey, saveCacheToDisk } = normalizeRunOptions(runOptions);
+    const { prefill, generationParams, cacheKey, ephemeral } = normalizeRunOptions(runOptions);
     const textMessages = [];
     const mediaItems = [];
     for (const message of prompt) {
@@ -77,7 +82,7 @@ function promptToAddonMessages(prompt, runOptions) {
         prefill,
         generationParams,
         cacheKey,
-        saveCacheToDisk,
+        ephemeral,
     });
     return promptMessages;
 }
@@ -97,8 +102,8 @@ const GENERATION_PARAM_KEYS = new Set([
     "grammar",
     "json_schema",
     "tool_choice",
+    "parallel_tool_calls",
     "reasoning_budget",
-    "remove_thinking_from_context",
 ]);
 // Normalizes the per-request `generationParams.json_schema` field. The
 // addon binding expects a string; callers commonly pass a plain object
@@ -137,14 +142,14 @@ function normalizeGenerationParams(generationParams) {
             params[key] = value;
     }
     const sanitized = params;
-    if (sanitized.remove_thinking_from_context !== undefined &&
-        typeof sanitized.remove_thinking_from_context !== "boolean") {
-        throw new TypeError("generationParams.remove_thinking_from_context must be a boolean when provided");
-    }
     if (sanitized.tool_choice !== undefined &&
         (typeof sanitized.tool_choice !== "string" || sanitized.tool_choice.length === 0)) {
         throw new TypeError('generationParams.tool_choice must be "auto", "none", "required" or a declared function name');
     }
+    if (sanitized.parallel_tool_calls !== undefined &&
+        typeof sanitized.parallel_tool_calls !== "boolean") {
+        throw new TypeError("generationParams.parallel_tool_calls must be a boolean when provided");
+    }
     const hasGrammar = typeof sanitized.grammar === "string" && sanitized.grammar.length > 0;
     const hasJsonSchema = sanitized.json_schema !== undefined &&
         sanitized.json_schema !== null &&
@@ -781,6 +786,46 @@ const LlmLlamacpp = class LlmLlamacpp {
     get _hasActiveResponse() {
         return (this.addon ? this.addon.activeJobs() : 0) > 0;
     }
+    /**
+     * Write the conversation kept in memory for `cacheKey` to that file now:
+     * the active single-prompt conversation, a batch conversation kept in its
+     * slot, or one in the RAM tier. Resolves once the file is written. When a
+     * request on that key is running it waits for it to finish, so the file
+     * always holds a committed state. A request still queued is not waited
+     * for: `run()` resolves when the request is accepted, so read its response
+     * to the end before saving when the file must hold that turn. Resolves without writing when the file
+     * already holds the conversation. Rejects when nothing is cached under the
+     * key and no file exists, or when the write fails (the conversation then
+     * stays in memory, still unsaved, so the call can be retried). Writes
+     * ephemeral conversations too.
+     */
+    async saveCache(cacheKey) {
+        if (typeof cacheKey !== "string" || cacheKey.length === 0) {
+            throw new TypeError("saveCache(cacheKey) requires a non-empty string");
+        }
+        if (!this.addon) {
+            throw new Error("Model is not loaded");
+        }
+        await this.addon.saveCache(cacheKey);
+    }
+    /**
+     * Drop the conversation kept in memory for `cacheKey` — the active
+     * single-prompt conversation, a batch conversation kept in its slot, its
+     * RAM-tier entry and its checkpoints — without writing it, so no later
+     * eviction or unload writes it. When a request on that key is running it
+     * waits for it to finish. The file, if one was written, is left alone:
+     * delete it as well to remove the conversation from disk. Resolves when
+     * nothing is kept for the key.
+     */
+    async discardCache(cacheKey) {
+        if (typeof cacheKey !== "string" || cacheKey.length === 0) {
+            throw new TypeError("discardCache(cacheKey) requires a non-empty string");
+        }
+        if (!this.addon) {
+            throw new Error("Model is not loaded");
+        }
+        await this.addon.discardCache(cacheKey);
+    }
     getState() {
         return this.state;
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@qvac/llm-llamacpp",
-  "version": "0.56.0",
+  "version": "0.58.0",
   "description": "llama addon for qvac",
   "addon": true,
   "scripts": {
@@ -78,7 +78,7 @@
     "@types/node": "^24.2.1",
     "bare-crypto": "^1.15.3",
     "bare-process": "^4.2.2",
-    "bare-subprocess": "^5.2.3",
+    "bare-subprocess": "^6.1.0",
     "bare-url": "^2.1.6",
     "brittle": "^3.16.5",
     "cmake-bare": "^1.7.5",
@@ -96,7 +96,7 @@
     "typescript-eslint": "^8.63.0"
   },
   "dependencies": {
-    "@qvac/fabric": "^0.20.0",
+    "@qvac/fabric": "^0.20.2",
     "@qvac/infer-base": "^0.6.2",
     "@qvac/logging": "^0.1.0",
     "bare-fs": "^4.5.1",