@qvac/llm-llamacpp 0.62.0 → 0.64.0

package/README.md

@@ -25,6 +25,7 @@ This native C++ addon, built using the `Bare` Runtime, simplifies running Large
 - [Architecture](#architecture)
 - [Benchmarking](#benchmarking)
 - [Tests](#tests)
+  - [Fuzzing](#fuzzing)
 - [Glossary](#glossary)
 - [License](#license)
 
@@ -210,6 +211,11 @@ const config = {
 | cache-type-k      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **key** quantization type. Unset = auto-default (see KV-cache type below) |
 | cache-type-v      | `f16`, `f32`, `bf16`, `q8_0`, `q4_0`, …      | auto (see below)             | KV-cache **value** quantization type. Quantizing V requires `flash-attn` on |
 | mmproj-use-gpu    | `"true"`/`"on"`/`"1"` or `"false"`/`"off"`/`"0"` | auto (see below)         | Run the multimodal projector (mmproj / vision encoder) on the GPU. Only honoured when a GPU backend is selected (ignored with a warning on CPU / GPU-fallback). Unset = auto-default (see mmproj backend below) |
+| spec-type         | `"none"`, `"draft-mtp"` or `"draft-dflash"` | `"none"`                     | Speculative decoding, as llama-server's `--spec-type`: `draft-mtp` drafts with the model's own multi-token-prediction head, `draft-dflash` with the DFlash model in `spec-draft-model` (see [Speculative decoding](#speculative-decoding-mtp-dflash) below). Also accepted as `spec_type` |
+| spec-draft-model  | absolute path                               | –                            | DFlash draft model, as llama-server's `--spec-draft-model` (`-md`). Required by, and only accepted with, `spec-type: "draft-dflash"`. Placed like the model (`device`, `gpu-layers`). Also accepted as `spec_draft_model` |
+| spec-draft-n-max  | 1 – 64                                      | 3                            | Longest draft per step. Also accepted as `spec_draft_n_max` |
+| spec-draft-n-min  | 0 – `spec-draft-n-max`                      | 0                            | Drafts shorter than this are not verified. Also accepted as `spec_draft_n_min` |
+| spec-draft-p-min  | 0 – 1                                       | 0                            | Minimum draft-head probability for a token to extend the draft. Also accepted as `spec_draft_p_min` |
 
 
 #### KV-cache type & auto-default
@@ -254,6 +260,21 @@ itself runs on the CPU backend (`device: "cpu"` or GPU fallback), the key is ign
 projector runs on CPU. The resolved choice is logged at verbosity ≥ 2 as
 `[LlamaModel] multimodal projector backend: …`.
 
+
+#### Speculative decoding (MTP, DFlash)
+
+`spec-type: 'draft-mtp'` turns on llama-server's multi-token-prediction speculative decoding for models that ship their MTP layers (`blk.N.nextn.*`, e.g. Qwen3.5 / Qwen3.6 GGUFs that keep them; quantizations that strip them fail to load with this option). Every step the MTP head drafts up to `spec-draft-n-max` tokens from the model's own hidden states, the model verifies the sampled token and the draft in one decode, and the longest matching prefix plus one new token is kept. The sampler decides every kept token, so sampling settings, grammars and stop conditions behave as without speculation. The implementation follows llama-server step by step: the same draft limits, rollback by trimming or by a checkpoint for memory that cannot trim, and the same replay.
+
+`spec-type: 'draft-dflash'` drafts with a separate DFlash model instead, given as `spec-draft-model` (an absolute path; llama-server's `-md`). The DFlash model reads the hidden states of the target layers it was trained on and denoises a whole block of draft tokens in one pass, so a step drafts up to its block size minus one tokens (`spec-draft-n-max` caps it further). It is loaded next to the model and placed like it (`device`, `gpu-layers`). The load's fit and `assessFit` pass it to fabric as an extra model, but fabric cannot measure a DFlash model yet: it logs `failed to measure the memory of the extra model, fitting without it`, so neither counts the draft model's memory. Leave room for it yourself. Everything else (verification, rollback, batching, caches, stats) is the same as for MTP.
+
+Both work on single prompts, multimodal prompts and continuous batching (`parallel >= 2`), together with `cacheKey` conversations: the draft context's state travels with checkpoints and the RAM tier, and a `cacheKey` file (target state only, like llama-server's slot files) restarts drafting from the next decoded token.
+
+- Runtime stats gain `draftTokens` and `draftAcceptedTokens` (llama-server's `draft_n` / `draft_n_accepted`); per-position acceptance rates are logged at debug level.
+- `spec-draft-n-max` is lowered at load, with a warning, so the sampled token and the draft fit one `ubatch-size` and each sequence's share of `batch-size` (`batch-size / parallel`); a batch too small for even one draft token fails the load.
+- `spec-draft-model` must be a regular local file.
+- Greedy output can differ from plain decoding where two tokens are nearly tied: the verification batch evaluates several positions at once, which changes floating-point rounding.
+- The gain depends on the device being bandwidth bound. On a Radeon 8060S, Qwen3.6-27B-MTP Q4_K_M decodes a single prompt at about 30 t/s instead of 13 t/s with 80% of the drafts accepted; with four concurrent sequences the larger verification batches cost more than they save, as they do in llama-server.
+
 ### 4. Create Model Instance
 
 ```js
@@ -678,6 +699,28 @@ npm run test:cpp:models:ci   # exactly what .github/workflows/cpp-tests-llm.yml
 
 First-run downloads pull several GB from Hugging Face. Every fixture is SHA256-verified against a digest pinned in `scripts/download-unit-test-models.js`; mismatched or partial files are re-downloaded automatically. Set `HF_TOKEN` if a repo requires authentication. Override paths with env vars such as `SHARDED_MODEL_FIRST_SHARD_PATH` (see `test/unit/test_common.hpp`).
 
+### Fuzzing
+
+`forLogMessage`, `toLowerAscii`, and `matchesAnyStopString` have a
+[Google FuzzTest][fuzztest] target, `llm-string-fuzz`. Fuzzing is Linux-only:
+it needs clang with libFuzzer and AddressSanitizer, CMake 3.25 or newer,
+[vcpkg](https://vcpkg.io/) (set `VCPKG_ROOT`) and `bare-make`. The target does
+not link `@qvac/fabric`, so it runs with full ASan and LeakSanitizer.
+
+```bash
+npm install
+npm run fuzz                     # bounded run of every FUZZ_TEST, as Linux CI does
+npm run fuzz:continuous          # coverage-guided, one FUZZ_TEST at a time
+npm run fuzz:continuous -- LlmStringFuzz.MatchesAnyStopNeverCrashes --fuzz_for=30m
+```
+
+Flags other than `--continuous` and `--build-dir` go to the fuzz binary. See
+[`docs/architecture/ADDON-FUZZING.md`](../../docs/architecture/ADDON-FUZZING.md)
+for the fuzzing design, the vcpkg-supplied FuzzTest stack, and how to add a
+target.
+
+[fuzztest]: https://github.com/google/fuzztest
+
 ## Glossary
 
 • **Bare Runtime** – Small and modular JavaScript runtime for desktop and mobile. [Learn more](https://docs.pears.com/reference/bare-overview).

package/index.d.ts

@@ -364,6 +364,41 @@ declare namespace LlmLlamacpp {
          * maximum is 1048576 (1 TiB). Also accepted as `cache-ram-mib`.
          */
         cache_ram_mib?: NumericLike;
+        /**
+         * Speculative decoding, as llama-server's `--spec-type`. `'draft-mtp'`
+         * drafts with the model's own multi-token-prediction (MTP / NextN) head;
+         * the model must ship its MTP layers (`blk.N.nextn.*`), or the load
+         * fails. `'draft-dflash'` drafts with the DFlash model given as
+         * `spec_draft_model`. Either way the model verifies the draft, so the
+         * output keeps the distribution of plain decoding. `'none'` (default)
+         * decodes one token per step. Works on single prompts, multimodal
+         * prompts and continuous batching. Also accepted as `spec-type`.
+         */
+        spec_type?: "none" | "draft-mtp" | "draft-dflash";
+        /**
+         * Absolute path of the DFlash draft model, as llama-server's
+         * `--spec-draft-model` (`-md`). Required by, and only accepted with,
+         * `spec_type: 'draft-dflash'`. The draft model is placed like the model
+         * (`device`, `gpu_layers`). Also accepted as `spec-draft-model`.
+         */
+        spec_draft_model?: string;
+        /**
+         * Longest draft per step (default 3, maximum 64), as
+         * `--spec-draft-n-max`. Requires `spec_type`. Also accepted as
+         * `spec-draft-n-max`.
+         */
+        spec_draft_n_max?: NumericLike;
+        /**
+         * Shortest draft worth verifying (default 0); shorter drafts are
+         * dropped. As `--spec-draft-n-min`. Also accepted as `spec-draft-n-min`.
+         */
+        spec_draft_n_min?: NumericLike;
+        /**
+         * Minimum draft-head probability for a token to extend the draft
+         * (default 0, between 0 and 1), as `--spec-draft-p-min`. Also accepted
+         * as `spec-draft-p-min`.
+         */
+        spec_draft_p_min?: NumericLike;
         [key: string]: string | number | boolean | string[] | undefined;
     }
     interface LlmLlamacppArgs {
@@ -599,6 +634,13 @@ declare namespace LlmLlamacpp {
          */
         avgConcurrentSeq: number;
         backendDevice: "cpu" | "gpu";
+        /**
+         * Draft tokens proposed by speculative decoding, and how many of them
+         * the model accepted (llama-server's `draft_n` / `draft_n_accepted`).
+         * Present only when the model was loaded with `spec_type`.
+         */
+        draftTokens?: number;
+        draftAcceptedTokens?: number;
         /**
          * Why generation stopped. Per-sequence, so it is reported for a single
          * request on either path (sequential or one prompt on a parallel model).

package/package.json

@@ -1,11 +1,11 @@
 {
   "name": "@qvac/llm-llamacpp",
-  "version": "0.62.0",
+  "version": "0.64.0",
   "description": "llama addon for qvac",
   "addon": true,
   "scripts": {
     "build": "npm run build:ts && npm run build:native",
-    "build:native": "bare-make generate && bare-make build && bare-make install",
+    "build:native": "bare-make generate -D BUILD_FUZZING=OFF && bare-make build && bare-make install",
     "build:ts": "tsc -p tsconfig.build.json",
     "build:pack": "npm run build:ts && mkdir -p dist && npm pack --pack-destination dist",
     "mobile:copy-prebuilds": "cp -r prebuilds/android-arm64 prebuilds/android-ia32 || echo 'Warning: Failed to copy llamacpp prebuilds to android-ia32'; cp -r prebuilds/android-arm64 prebuilds/android-arm || echo 'Warning: Failed to copy llamacpp prebuilds to android-arm'; cp -r prebuilds/android-arm64 prebuilds/android-x64 || echo 'Warning: Failed to copy llamacpp prebuilds to android-x64'; cp -r prebuilds/ios-arm64 prebuilds/ios-arm64-simulator 2>/dev/null || echo 'iOS prebuilds already present'; cp -r prebuilds/ios-arm64 prebuilds/ios-x64-simulator 2>/dev/null || echo 'iOS prebuilds already present'",
@@ -32,13 +32,13 @@
     "test:unit:generate": "brittle -r test/unit/all.js test/unit/*.test.js",
     "test:prestage": "node --test scripts/__tests__/*.test.js ../../.github/actions/cache-models/warm-models.test.mjs",
     "test:unit": "npm run build:ts && npm run test:unit:generate && bare test/unit/all.js --exit && npm run test:prestage && npm run test:mobile:validate",
-    "test:cpp:build": "bare-make generate -D BUILD_TESTING=ON && bare-make build --target addon-test",
+    "test:cpp:build": "bare-make generate -D BUILD_TESTING=ON -D BUILD_FUZZING=OFF && bare-make build --target addon-test",
     "test:cpp:models": "node scripts/download-unit-test-models.js",
     "test:cpp:models:ci": "node scripts/download-unit-test-models.js --ci",
     "test:cpp:run": "node scripts/run-cpp-tests.js",
     "test:cpp:run:ci": "node scripts/run-cpp-tests.js --ci",
     "test:cpp": "npm run test:cpp:build && npm run test:cpp:run",
-    "coverage:cpp:build": "bare-make generate -D BUILD_TESTING=ON -D ENABLE_COVERAGE=ON && bare-make build --target addon-test",
+    "coverage:cpp:build": "bare-make generate -D BUILD_TESTING=ON -D ENABLE_COVERAGE=ON -D BUILD_FUZZING=OFF && bare-make build --target addon-test",
     "coverage:cpp:run": "node scripts/run-cpp-tests.js --coverage",
     "coverage:cpp:run:ci": "node scripts/run-cpp-tests.js --coverage --ci",
     "coverage:cpp:summary": "cd build/test/unit && llvm-cov report ./addon-test --instr-profile=coverage.profdata -ignore-filename-regex='(tests|build|node_modules|gtest|gmock|\\.vcpkg|/usr)/' > coverage-summary.txt",
@@ -46,7 +46,13 @@
     "coverage:cpp": "npm run coverage:cpp:build && npm run coverage:cpp:run && npm run coverage:cpp:report",
     "test:all": "npm run test:unit && npm run test:integration && npm run test:cpp",
     "benchmarks": "./benchmarks/run-benchmarks.sh",
-    "benchmarks:windows": "powershell -ExecutionPolicy Bypass -File ./benchmarks/run-benchmarks.ps1"
+    "benchmarks:windows": "powershell -ExecutionPolicy Bypass -File ./benchmarks/run-benchmarks.ps1",
+    "fuzz:build": "bare-make generate -D BUILD_FUZZING=ON -D BUILD_TESTING=OFF -D FUZZTEST_FUZZING_MODE=OFF && bare-make build --target llm-string-fuzz",
+    "fuzz:build:continuous": "bare-make generate -D BUILD_FUZZING=ON -D BUILD_TESTING=OFF -D FUZZTEST_FUZZING_MODE=ON && bare-make build --target llm-string-fuzz",
+    "fuzz:run": "npm run test:scripts && node scripts/run-cpp-fuzz.js",
+    "test:scripts": "node --test scripts/__tests__/run-cpp-fuzz.test.js",
+    "fuzz": "npm run fuzz:build && npm run fuzz:run",
+    "fuzz:continuous": "npm run fuzz:build:continuous && node scripts/run-cpp-fuzz.js --continuous"
   },
   "files": [
     "binding.js",
@@ -73,8 +79,8 @@
   "bugs": "https://github.com/tetherto/qvac/issues",
   "homepage": "https://qvac.tether.io",
   "devDependencies": {
-    "@qvac/fabric-android-arm64": "0.22.0",
-    "@qvac/fabric-ios": "0.22.0",
+    "@qvac/fabric-android-arm64": "0.23.0",
+    "@qvac/fabric-ios": "0.23.0",
     "@types/node": "^24.2.1",
     "bare-crypto": "^1.15.3",
     "bare-process": "^4.2.2",
@@ -96,22 +102,22 @@
     "typescript-eslint": "^8.63.0"
   },
   "dependencies": {
-    "@qvac/fabric": "^0.22.0",
+    "@qvac/fabric": "^0.23.0",
     "@qvac/infer-base": "^0.8.0",
     "@qvac/logging": "^0.1.0",
     "bare-fs": "^4.5.1",
     "bare-path": "^3.0.0"
   },
   "optionalDependencies": {
-    "@qvac/fabric-linux-x64": "^0.22.0",
-    "@qvac/fabric-linux-arm64": "^0.22.0",
-    "@qvac/fabric-darwin-arm64": "^0.22.0",
-    "@qvac/fabric-darwin-x64": "^0.22.0",
-    "@qvac/fabric-win32-x64": "^0.22.0"
+    "@qvac/fabric-linux-x64": "^0.23.0",
+    "@qvac/fabric-linux-arm64": "^0.23.0",
+    "@qvac/fabric-darwin-arm64": "^0.23.0",
+    "@qvac/fabric-darwin-x64": "^0.23.0",
+    "@qvac/fabric-win32-x64": "^0.23.0"
   },
   "peerDependencies": {
-    "@qvac/fabric-android-arm64": "^0.22.0",
-    "@qvac/fabric-ios": "^0.22.0"
+    "@qvac/fabric-android-arm64": "^0.23.0",
+    "@qvac/fabric-ios": "^0.23.0"
   },
   "peerDependenciesMeta": {
     "@qvac/fabric-android-arm64": {