xinfer-ai 0.13.6 → 0.13.8

package/README.md

@@ -329,7 +329,7 @@ xinfer --m mistralai/Ministral-3-3B --ui-server
 ```bash
 # Prerequisites: Rust compiler, CUDA Toolkit (optional) or Metal Xcode command line tool
 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-sudo apt-get install -y git build-essential libssl-dev pkg-config
+sudo apt-get install -y git build-essential
 
 export XINFER_REPO="https://github.com/guoqingbao/xinfer"
 # MacOS/Metal: replace features to `metal`
@@ -449,9 +449,14 @@ xinfer --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF \
 
 ---
 
-## 🔌 Structured Outputs
+## 🔌 Structured Outputs & Guided Decoding
 
-Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema.
+Constraint-based generation via llguidance — JSON Schema, regex, Lark grammars, choice lists.
+
+```bash
+# Enable structured outputs and tool grammar
+xinfer --m Qwen/Qwen3.6-27B-FP8 --ui-server --enable-tool-grammar
+```
 
 [Structured outputs documentation →](docs/guided_decoding.md)
 
@@ -495,7 +500,7 @@ Constraint-based generation via llguidance — Lark grammars, regex, JSON Schema
 | `--max-num-seqs` | Max concurrent requests (default: 32, macOS: 8) |
 | `--max-tokens` | Max tokens per response (default: 16384) |
 | `--kv-fraction` | GPU memory fraction for KV cache |
-| `--cpu-mem-fold` | CPU swap memory ratio (default: 0.2) |
+| `--cpu-mem-fold` | CPU swap memory ratio (default: 0.5) |
 | `--pd-server` | Run as PD prefill server |
 | `--pd-client` | Run as PD decode client |
 | `--pd-url` | PD connection URL (`tcp://`, `http://`, `file://`) |

package/docs/guided_decoding.md

@@ -8,13 +8,23 @@ It focuses on:
 - how reasoning effort is applied
 - practical usage and validation commands
 
+## JSON Schema Reference
+
+For detailed JSON Schema constraint documentation with curl examples, see [`llguidance-json-schema.md`](llguidance-json-schema.md).
+
+This covers:
+- Schema type definitions (string, integer, number, boolean, object, array)
+- All supported API endpoints (OpenAI-compatible and Claude server)
+- Complete curl examples for each permutation
+- Schema sanitization behavior
+
 ## Current Model
 
 Guided decoding is request-scoped.
 
 The core engine does not invent grammars on its own. A request either:
 - supplies a constraint grammar
-- gets that constraint grammar composed with a reasoning prefix
+- gets a composed grammar containing both
 - or runs unconstrained when neither exists
 
 The final grammar is stored in `SamplingParams.grammar` and consumed by the runner.
@@ -44,7 +54,7 @@ The server composes:
 
 The result is a single `TopLevelGrammar` assigned to `SamplingParams.grammar`.
 
-If no client-supplied constraint grammar exists, `params.grammar` stays `None`.
+If no constraint grammar and no tool grammar exist, `params.grammar` stays `None`.
 
 ### 3. Sampling in runner
 
@@ -84,14 +94,17 @@ Legacy fields
 - `constraint`
 - `constraint_type = regex | lark | json_schema | json`
 
-If a request provides none of the above, guided decoding is not enabled.
+If a request provides none of the above, guided decoding is not enabled unless tool grammar synthesis adds one.
+
+The grammar composition logic is in `src/utils/guidance_grammar.rs`. The `GrammarRequestDispatcher` and `GrammarComposer` handle the composition of constraint grammars with reasoning grammars.
 
 ### Claude server
 
+Claude reuses the same tool-grammar builder path.
+
 Current state:
 - Claude does not expose the same client-supplied grammar request surface as the OpenAI endpoint
 - Claude reasoning is still driven by explicit thinking behavior, not by `reasoning_effort` grammar composition
-- Claude requests therefore do not currently enable guided decoding
 
 ## Reasoning Effort
 
@@ -99,7 +112,7 @@ Reasoning effort is separate from ordinary structured outputs.
 
 ### Current state
 
-The OpenAI path maps `reasoning_effort` into grammar composition when a request constraint exists.
+The OpenAI path maps `reasoning_effort` into grammar composition.
 
 Accepted values come from `ReasoningEffort::from_str`:
 - `none`
@@ -115,7 +128,7 @@ Non-Python builds also support:
 - `custom:<template>`
 
 Relevant code:
-- `src/utils/reasoning.rs`
+- `src/utils/guidance_grammar.rs`
 - `src/server/server.rs`
 - `src/utils/guidance.rs`
 
@@ -124,7 +137,6 @@ Relevant code:
 Reasoning effort:
 - does not enable chat-template thinking by itself
 - only affects grammar composition
-- is ignored when no request constraint grammar is present
 - only works when reasoning start/end tokens are available
 
 If the tokenizer does not expose reasoning markers, the system logs a warning and falls back to the base grammar.
@@ -160,7 +172,7 @@ xinfer --m Qwen/Qwen3.5-35B-A3B-FP8/ --ui-server --d 0
 | Enforce text pattern | `structured_outputs` or `constraint` | `regex` |
 | Enforce full object schema | `structured_outputs` or `response_format` | `json` / `json_schema` |
 | Enforce custom grammar | `structured_outputs` or `constraint` | `grammar` / `lark` |
-| Constrain tagged structured output payload | `structured_outputs` | `structural_tag` |
+| Constrain tool call payload | `structured_outputs` or automatic tool grammar | `structural_tag` / tool grammar |
 | Add a reasoning prefix | `reasoning_effort` | `low`, `medium`, `high`, etc. |
 
 ### 1. Constrain the answer to a fixed set
@@ -470,5 +482,5 @@ Check:
 
 - Guided decoding is only active when `SamplingParams.grammar` is present.
 - OpenAI currently has the richest client-facing grammar surface.
-- Claude does not currently expose request-level guided decoding.
+- Claude currently reuses tool grammar, but not the same direct constraint request API.
 - No request-level grammar means no guided decoding.

package/docs/prefix-cache.md

@@ -46,6 +46,31 @@ snapshot capture remains dense.
 - Cached KV reuse is automatic; no `session_id` is required.
 - Sliding window attention limits how much cached context is effectively used.
 
+## CPU swap with prefix cache
+
+When prefix cache is enabled, live sequence preemption is partial:
+
+- Leading prefix/shared blocks stay on GPU and keep their sequence reference.
+- Only the sequence-owned suffix blocks are copied to CPU swap.
+- Swap-in reuses the retained GPU prefix blocks, allocates new suffix blocks,
+  and copies only the suffix KV back from CPU.
+
+This is different from prefix-cache offload. Prefix-cache offload only helps when
+an evicted cached prefix is requested again later. Partial sequence swap helps
+immediately under memory pressure because it avoids recomputing the active
+sequence suffix after preemption, even when no future request reuses that suffix.
+For hybrid Mamba/GDN models, the active recurrent state remains tied to the
+swapped sequence lifecycle and is not released until the sequence finishes or is
+cancelled. On CUDA, Mamba/GDN prefix snapshots use a separate two-tier cache:
+device snapshots are the fast path, and snapshots evicted from device memory are
+spilled to CPU so a later KV prefix hit can promote the matching recurrent state
+back instead of falling back to a shorter partial hit. Metal uses unified memory,
+so this CPU spill tier is disabled there. The CUDA CPU snapshot tier is 4x the
+device snapshot capacity. Prompt/prefill snapshots and final decode-boundary
+snapshots are protected from ordinary decode-time snapshot churn. When the CPU
+snapshot tier is full, LRU eviction frees at least 10% of the tier in one batch
+before accepting new GDN/Mamba snapshot offloads.
+
 ## Inspecting cache hits
 
 Chat completion responses include the prefix-cache hit count under

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "xinfer-ai",
-  "version": "0.13.6",
-  "assetVersion": "0.13.6",
+  "version": "0.13.8",
+  "assetVersion": "0.13.8",
   "description": "xInfer — a high-performance LLM inference engine in Rust with CUDA/Metal acceleration",
   "license": "MIT",
   "homepage": "https://github.com/guoqingbao/xinfer#readme",