@lloyal-labs/lloyal.node 1.0.6-alpha → 1.0.8

package/README.md

@@ -1,14 +1,68 @@
 # lloyal.node
 
-**Advanced edge inference for Node.js**
+**Covalent inference for Node.js**
 
-Inference with forkable state — KV cache, grammar, metrics all clone atomically. Entropy and surprisal mid-generation. Multi-sequence parallel exploration. The control surface llama.cpp exposes, in TypeScript.
+Forkable inference state for llama.cpp — Branch a generation into a tree — prefix sharing is the bond across branches while each owns its own machinery (sampler chain, seed, grammar, logits snapshot, perplexity tracker) enabling controlled divergence at decode time.
+
+## Branch API
+
+Fork from root for best-of-N, fork from children for MCTS/beam search, fork from a draft for speculative decoding. The produce/commit protocol separates sampling from state advancement — sample without writing to KV, inspect the result, then decide whether to commit.
+
+```javascript
+import { createContext, Branch } from '@lloyal-labs/lloyal.node';
+
+const ctx = await createContext({ modelPath: './model.gguf', nSeqMax: 8 });
+const tokens = await ctx.tokenize('Once upon a time');
+await ctx.decode(tokens, 0, 0);
+
+// Create root branch, freeze logits from prefill
+const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
+root.captureLogits();
+
+// Fork N candidates — KV prefix shared, sampler/grammar/logits/perplexity cloned
+const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
+  const branch = root.fork(seqId);
+  branch.reseedSampler(1000 + i);
+  return branch;
+});
+
+// Generate (interleaved round-robin)
+for (let t = 0; t < 50; t++) {
+  for (const branch of candidates) {
+    const { token, isStop } = branch.produce(); // Sample, no KV write
+    if (isStop) continue;
+    branch.commit(token); // Accept + forward pass + capture
+  }
+}
+
+// Select by perplexity, prune losers
+const best = candidates.reduce((a, b) => (a.perplexity < b.perplexity ? a : b));
+for (const c of candidates) {
+  if (c !== best) c.prune();
+}
+```
+
+**What `fork()` shares:** KV cache prefix (metadata-only under unified KV — no tensor buffers copied).
+
+**What `fork()` clones:** Logits snapshot, sampler chain (penalties + PRNG), grammar state, logit bias, perplexity tracker.
+
+**Key methods:**
+
+- `produce()` / `commit()` — two-phase: sample without KV write, then advance
+- `prune()` — discard loser and its divergent KV entries
+- `destroy()` — release handle, keep KV (for winners continuing with raw ops)
+- `reseedSampler()` — unique PRNG per fork for stochastic diversity
+- `perplexity` — rolling PPL per branch for quality-based selection
+
+---
+
+## Install
 
 ```bash
 npm install @lloyal-labs/lloyal.node
 ```
 
-Prebuilt binaries for 13 platforms:
+Prebuilt binaries for 13 platform/GPU combinations. GPU selection at runtime, not install time.
 
 | Platform | Arch  | Acceleration        |
 | -------- | ----- | ------------------- |
@@ -19,80 +73,32 @@ Prebuilt binaries for 13 platforms:
 | Windows  | x64   | CPU / CUDA / Vulkan |
 | Windows  | arm64 | CPU / Vulkan        |
 
-GPU selection happens at runtime, not install time. See [distribution.md](docs/distribution.md) for details.
+See [distribution.md](docs/distribution.md) for details.
 
 ---
 
 ## Examples
 
-Working examples demonstrate each capability:
-
-| Example                                   | What It Demonstrates                                                          |
-| ----------------------------------------- | ----------------------------------------------------------------------------- |
-| [`best-of-n/`](./examples/best-of-n/)     | Multi-sequence generation, PPL selection, captured logits for fair comparison |
-| [`speculative/`](./examples/speculative/) | KV forking, draft/verify/accept/reject, `kvCacheRemove` for rejected tokens   |
-| [`entropy/`](./examples/entropy/)         | Entropy Decision Tree — `modelEntropy()` mid-generation as control signal     |
-| [`grammar/`](./examples/grammar/)         | Pull loop with generators, JSON schema constraints, KV + grammar branching    |
-| [`streaming/`](./examples/streaming/)     | Infinite context via BlinkKV, `clearAndReseed`, perplexity tracking           |
-| [`chat/`](./examples/chat/)               | Interactive streaming chat                                                    |
-| [`embed/`](./examples/embed/)             | Text embeddings extraction                                                    |
+| Example                                   | Pattern                                                                    |
+| ----------------------------------------- | -------------------------------------------------------------------------- |
+| [`best-of-n/`](./examples/best-of-n/)     | Branch API: fork, produce/commit, perplexity selection                     |
+| [`speculative/`](./examples/speculative/) | Branch API: draft/verify, fork/prune, bonus token sampling                 |
+| [`streaming/`](./examples/streaming/)     | Infinite context via BlinkKV reseeding with sidecar summarization          |
+| [`entropy/`](./examples/entropy/)         | `modelEntropy()` mid-generation as control signal                          |
+| [`grammar/`](./examples/grammar/)         | Pull loop with generators, JSON schema constraints, KV + grammar branching |
+| [`chat/`](./examples/chat/)               | Interactive streaming chat                                                 |
+| [`embed/`](./examples/embed/)             | Text embeddings extraction                                                 |
 
 ```bash
 node examples/best-of-n/best-of-n.mjs
 node examples/speculative/speculative.mjs
-node examples/entropy/entropy.mjs
-node examples/grammar/grammar.mjs
 ```
 
-Each example has a README explaining the pattern in depth.
+Each example has a README explaining the pattern.
 
 ---
 
-## Core Patterns
-
-### Forkable State
-
-KV cache, grammar parser, and perplexity trackers all live behind handles. Handles clone atomically.
-
-**Two forking strategies:**
-
-| Approach             | Method                            | Use Case                                     |
-| -------------------- | --------------------------------- | -------------------------------------------- |
-| **Tag copy**         | `kvSeqCopy(src, dst)`             | Parallel branches with different seqIds      |
-| **Snapshot/restore** | `kvCacheSave()` / `kvCacheLoad()` | Sequential exploration, return to checkpoint |
-
-[`examples/best-of-n/`](./examples/best-of-n/) uses tag copy — each candidate gets its own seqId, branches run in parallel:
-
-```javascript
-ctx.kvSeqCopy(0, seqId); // O(1) tag copy, branch diverges on seqId
-```
-
-[`examples/grammar/`](./examples/grammar/) uses snapshot/restore — save state, explore branches sequentially, restore between each:
-
-```javascript
-const snapshot = await ctx.kvCacheSave(0); // Save checkpoint
-// ... explore branch ...
-await ctx.kvCacheLoad(0, snapshot); // Return to checkpoint
-```
-
-Both approaches also fork grammar state with `cloneSampler()` when grammar constraints are involved.
-
-### Captured Logits
-
-After decode, logits represent P(next_token | context). When forking to multiple sequences, capture logits for fair comparison:
-
-```javascript
-// Capture after prefill
-const capturedLogits = new Float32Array(ctx.getLogits());
-
-// All candidates sample first token from same distribution
-const token = sampleWithStrategy(capturedLogits, { params, workspace, prng });
-
-// Compute surprisal from captured logits (native C++)
-const surprisal = ctx.modelSurprisal(token, 'nats', capturedLogits);
-```
-
-See [`examples/best-of-n/`](./examples/best-of-n/) for the full pattern.
+## Other Patterns
 
 ### Entropy as Control Signal
 
@@ -109,26 +115,21 @@ if (entropy > 4.0) {
 
 See [`examples/entropy/`](./examples/entropy/) for entropy-triggered sampling strategies.
 
-### Pull Loop with Generators
+### Low-Level KV Operations
 
-For branching mid-generation, generators provide natural backpressure:
+For fine-grained control without Branch:
 
-```javascript
-function* tokenGenerator(ctx, grammarHandle) {
-  while (true) {
-    const logits = ctx.getLogits();
-    ctx.applySampler(grammarHandle, logits);
-    const token = ctx.sample({ temperature: 0.7 });
-    if (ctx.isStopToken(token)) return;
-    ctx.acceptSamplerToken(grammarHandle, token);
-    yield { token, text: ctx.tokenToText(token) };
-  }
-}
+| Approach             | Method                            | Use Case                                     |
+| -------------------- | --------------------------------- | -------------------------------------------- |
+| **Sequence copy**    | `kvSeqCopy(src, dst)`             | Share prefix across sequences                |
+| **Snapshot/restore** | `kvCacheSave()` / `kvCacheLoad()` | Sequential exploration, return to checkpoint |
 
-// Consumer controls pace — stop at branch point
-for (const { token, text } of gen) {
-  if (accumulated.includes('"city"')) break; // Pause here, branch
-}
+### Grammar-Constrained Generation
+
+```javascript
+const grammar = ctx.jsonSchemaToGrammar(schema);
+const handle = ctx.createSampler(grammar);
+// Pull loop — consumer controls pace, can branch at any point
 ```
 
 See [`examples/grammar/`](./examples/grammar/) for the full pull loop pattern.
@@ -137,80 +138,9 @@ See [`examples/grammar/`](./examples/grammar/) for the full pull loop pattern.
 
 ## API Reference
 
-### Context Creation
-
-```typescript
-const ctx = await createContext({
-  modelPath: string,       // Path to .gguf file (required)
-  nCtx?: number,           // Context size (default: 2048)
-  nThreads?: number,       // CPU threads (default: 4)
-  embeddings?: boolean,    // Enable embedding mode (default: false)
-  poolingType?: number,    // 0=NONE, 1=MEAN, 2=CLS, 3=LAST
-  nSeqMax?: number,        // Max parallel sequences (default: 1)
-});
-```
+Full API documentation: **[lloyal-ai.github.io/lloyal.node](https://lloyal-ai.github.io/lloyal.node/)**
 
-### Core Methods
-
-| Method                        | Returns             | Description                     |
-| ----------------------------- | ------------------- | ------------------------------- |
-| `tokenize(text)`              | `Promise<number[]>` | Text → token IDs                |
-| `detokenize(tokens)`          | `Promise<string>`   | Token IDs → text                |
-| `tokenToText(token)`          | `string`            | Single token → text (streaming) |
-| `decode(tokens, pos, seqId?)` | `Promise<void>`     | Forward pass, updates KV cache  |
-| `sample(params?)`             | `number`            | Sample next token               |
-| `isStopToken(token)`          | `boolean`           | Check for EOS token             |
-| `getLogits()`                 | `Float32Array`      | Raw logits (zero-copy view)     |
-
-### KV Cache
-
-| Method                             | Returns           | Description                    |
-| ---------------------------------- | ----------------- | ------------------------------ |
-| `kvCacheSize(seqId?)`              | `number`          | Tokens in cache                |
-| `kvCacheClear()`                   | `Promise<void>`   | Clear all sequences            |
-| `kvCacheRemove(seqId, start, end)` | `Promise<void>`   | Remove token range             |
-| `kvCacheSave(seqId?)`              | `Promise<Buffer>` | Snapshot state                 |
-| `kvCacheLoad(seqId, state)`        | `Promise<void>`   | Restore state                  |
-| `kvSeqCopy(src, dst)`              | `void`            | Copy sequence (tag copy, O(1)) |
-| `kvSeqKeep(seqId)`                 | `void`            | Keep only one sequence         |
-| `clearAndReseed(sinks, tail)`      | `Promise<void>`   | BlinkKV pattern                |
-
-### Grammar (Handle-Based)
-
-| Method                           | Returns  | Description                 |
-| -------------------------------- | -------- | --------------------------- |
-| `jsonSchemaToGrammar(schema)`    | `string` | Schema → GBNF               |
-| `createSampler(grammarStr)`      | `number` | Create grammar handle       |
-| `cloneSampler(handle)`           | `number` | Clone grammar state         |
-| `applySampler(handle, logits)`   | `void`   | Apply constraints to logits |
-| `acceptSamplerToken(handle, id)` | `void`   | Advance parser state        |
-| `freeSamplerHandle(handle)`      | `void`   | Release grammar handle      |
-
-### Metrics
-
-| Method                                  | Returns         | Description                                |
-| --------------------------------------- | --------------- | ------------------------------------------ |
-| `modelEntropy(base?, logits?)`          | `number`        | Distribution entropy (bits/nats)           |
-| `modelSurprisal(token, base?, logits?)` | `number`        | Token surprisal (supports captured logits) |
-| `createPerplexityTracker()`             | `TrackerHandle` | Create tracker (forkable)                  |
-| `clonePerplexityTracker(handle)`        | `TrackerHandle` | Clone tracker state                        |
-| `addSurprisal(handle, value)`           | `void`          | Add to tracker                             |
-| `getPerplexity(handle)`                 | `number`        | Get current PPL                            |
-| `freePerplexityTracker(handle)`         | `void`          | Release tracker                            |
-
-### Embeddings
-
-| Method                      | Returns         | Description                 |
-| --------------------------- | --------------- | --------------------------- |
-| `encode(tokens)`            | `Promise<void>` | Forward pass for embeddings |
-| `getEmbeddings(normalize?)` | `Float32Array`  | Extract embedding vector    |
-| `getEmbeddingDimension()`   | `number`        | Vector dimension            |
-
-### Lifecycle
-
-| Method      | Description                          |
-| ----------- | ------------------------------------ |
-| `dispose()` | Free native resources (**required**) |
+Generated from [`lib/index.d.ts`](./lib/index.d.ts) with TypeDoc.
 
 ---
 

package/lib/Branch.js

@@ -0,0 +1,268 @@
+/**
+ * Branch - Forkable inference handle for covalent generation
+ *
+ * A Branch owns everything needed for independent generation: a KV cache
+ * sequence, sampler chain, logits snapshot, and perplexity tracker.
+ *
+ * Forking is cheap — the KV prefix is shared in memory (metadata-only operation under unified KV —
+ * no KV tensor buffers are copied), so sibling branches read from the same physical KV entries.
+ * Only tokens decoded after the fork point are exclusive to each branch.
+ * This is the covalent property: branches share a bond (common prefix)
+ * while diverging independently.
+ *
+ * Branches form trees, not just flat lists. Fork from root for best-of-N,
+ * fork from children for MCTS/beam search, fork from a draft for speculative
+ * decoding.
+ *
+ * The produce/commit protocol separates sampling from state advancement:
+ * produce() samples without writing to KV, letting you inspect the result
+ * before deciding to commit(). This two-phase split is what makes speculative
+ * verification and tree search natural.
+ *
+ * @example Best-of-N with perplexity selection
+ * ```js
+ * const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
+ * root.captureLogits();
+ *
+ * const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
+ *   const branch = root.fork(seqId);
+ *   branch.reseedSampler(1000 + i);
+ *   return branch;
+ * });
+ *
+ * for (let t = 0; t < 50; t++) {
+ *   for (const branch of candidates) {
+ *     const { token, isStop } = branch.produce();
+ *     if (isStop) continue;
+ *     branch.commit(token);
+ *   }
+ * }
+ *
+ * const best = candidates.reduce((a, b) => a.perplexity < b.perplexity ? a : b);
+ * for (const c of candidates) { if (c !== best) c.prune(); }
+ * ```
+ */
+
+class Branch {
+  /**
+   * @param {SessionContext} ctx
+   * @param {number} handle
+   */
+  constructor(ctx, handle) {
+    this._ctx = ctx;
+    this._handle = handle;
+    this._disposed = false;
+  }
+
+  /**
+   * Create a root branch at the given position
+   *
+   * The branch takes ownership of the sequence and creates its own sampler
+   * chain from the provided params. Call captureLogits() after prefill to
+   * freeze the logit distribution before forking.
+   *
+   * @param {SessionContext} ctx - SessionContext to create branch on
+   * @param {number} seqId - Sequence ID for this branch
+   * @param {number} position - Starting position (typically prompt token count)
+   * @param {SamplingParams} [params] - Sampling parameters (temperature, topP, etc.)
+   * @returns {Branch} New Branch instance
+   */
+  static create(ctx, seqId, position, params) {
+    const handle = ctx._branchCreate(seqId, position, params);
+    return new Branch(ctx, handle);
+  }
+
+  /**
+   * Fork this branch to a new sequence
+   *
+   * The child shares the parent's KV prefix in memory (metadata-only under unified KV, no KV buffer copy).
+   * Logits, sampler state, and perplexity tracker are cloned so the child
+   * can diverge independently. Fork from any branch — root or intermediate —
+   * to build arbitrarily deep trees.
+   *
+   * Call reseedSampler() on each child for stochastic diversity.
+   *
+   * @param {number} newSeqId - Sequence ID for the forked branch
+   * @returns {Branch} New forked Branch
+   */
+  fork(newSeqId) {
+    this._ensureNotDisposed();
+    const newHandle = this._ctx._branchFork(this._handle, newSeqId);
+    return new Branch(this._ctx, newHandle);
+  }
+
+  /**
+   * Freeze the current logit distribution into this branch
+   *
+   * Logits are ephemeral — they're overwritten on the next decode() call.
+   * Capturing preserves them so this branch (and any forks from it) can
+   * sample from the same distribution. Essential before fork().
+   */
+  captureLogits() {
+    this._ensureNotDisposed();
+    this._ctx._branchCaptureLogits(this._handle);
+  }
+
+  /**
+   * Single-token forward pass with logit snapshot
+   *
+   * Runs one decode step (writing the token's KV entries), advances position,
+   * and captures the resulting logits for the next sample() call.
+   *
+   * @param {number} token - Token to decode
+   */
+  decodeAndCaptureOne(token) {
+    this._ensureNotDisposed();
+    this._ctx._branchDecodeAndCaptureOne(this._handle, token);
+  }
+
+  /**
+   * Sample next token from branch's logits snapshot
+   *
+   * Applies the branch's full sampler chain (top-k, top-p, temperature,
+   * repeat/presence penalties) to the captured logits.
+   *
+   * @returns {number} Sampled token ID
+   */
+  sample() {
+    this._ensureNotDisposed();
+    return this._ctx._branchSample(this._handle);
+  }
+
+  /**
+   * Record token in the sampler's repeat/presence penalty window
+   *
+   * @param {number} token - Token to accept
+   */
+  accept(token) {
+    this._ensureNotDisposed();
+    this._ctx._branchAccept(this._handle, token);
+  }
+
+  /**
+   * Discard this branch entirely — remove its KV entries and free the handle
+   *
+   * Use for losers: branches whose generation you want to erase completely.
+   * Only removes KV entries divergent from the shared prefix; sibling
+   * branches are unaffected.
+   */
+  prune() {
+    if (this._disposed) return;
+    this._ctx._branchPrune(this._handle);
+    this._disposed = true;
+  }
+
+  /**
+   * Release the handle but keep KV cache entries intact
+   *
+   * Use for winners: you're done branching but want to continue generation
+   * on this sequence using raw ctx.decode()/ctx.sample() calls. The KV
+   * cache entries remain at their current positions.
+   */
+  destroy() {
+    if (this._disposed) return;
+    this._ctx._branchDestroy(this._handle);
+    this._disposed = true;
+  }
+
+  /**
+   * Reseed the sampler's PRNG for diversity after fork()
+   *
+   * CRITICAL for parallel generation: Without reseeding, all forked branches
+   * produce identical outputs because they share the same PRNG state.
+   *
+   * Only affects stochastic samplers (temperature > 0). Greedy samplers are unchanged.
+   *
+   * @param {number} seed - New seed for the PRNG
+   *
+   * @example
+   * ```js
+   * const root = Branch.create(ctx, 0, pos, { temperature: 0.9 });
+   * root.captureLogits();
+   *
+   * // Fork and reseed for diversity
+   * const branches = [1, 2, 3, 4, 5].map((seqId, i) => {
+   *   const branch = root.fork(seqId);
+   *   branch.reseedSampler(1000 + i);  // Each branch gets unique seed
+   *   return branch;
+   * });
+   * ```
+   */
+  reseedSampler(seed) {
+    this._ensureNotDisposed();
+    this._ctx._branchSamplerChainReseed(this._handle, seed);
+  }
+
+  /**
+   * Sample the next token without advancing state
+   *
+   * No KV write, no position update. Inspect the result before deciding
+   * to commit() — this separation is what enables speculative verification
+   * and conditional branching.
+   *
+   * @returns {{ token: number, text: string, isStop: boolean }}
+   */
+  produce() {
+    this._ensureNotDisposed();
+    const token = this.sample();
+    return {
+      token,
+      text: this._ctx.tokenToText(token),
+      isStop: this._ctx.isStopToken(token),
+    };
+  }
+
+  /**
+   * Accept and advance — write token to KV and update branch state
+   *
+   * Accepts the token for repeat-penalty tracking, decodes it (writing to
+   * KV cache), and captures the resulting logits for the next produce() call.
+   *
+   * @param {number} token - Token to commit (from produce())
+   */
+  commit(token) {
+    this._ensureNotDisposed();
+    this.accept(token);
+    this.decodeAndCaptureOne(token);
+  }
+
+  // ===== ACCESSORS =====
+
+  /** @returns {number} Branch's sequence ID */
+  get seqId() {
+    this._ensureNotDisposed();
+    return this._ctx._branchGetSeqId(this._handle);
+  }
+
+  /** @returns {number} Branch's current position (number of tokens decoded) */
+  get position() {
+    this._ensureNotDisposed();
+    return this._ctx._branchGetPosition(this._handle);
+  }
+
+  /** @returns {number} Branch's perplexity (exp of mean surprisal) */
+  get perplexity() {
+    this._ensureNotDisposed();
+    return this._ctx._branchGetPerplexity(this._handle);
+  }
+
+  /** @returns {number} Internal handle (for debugging) */
+  get handle() {
+    return this._handle;
+  }
+
+  /** @returns {boolean} Whether this branch has been disposed */
+  get disposed() {
+    return this._disposed;
+  }
+
+  // ===== INTERNAL =====
+
+  _ensureNotDisposed() {
+    if (this._disposed) {
+      throw new Error('Branch has been disposed');
+    }
+  }
+}
+
+module.exports = { Branch };

package/lib/index.d.ts

@@ -347,9 +347,11 @@ export interface SessionContext {
    * // Creative generation
    * const token = ctx.sample({ temperature: 0.9 });
    *
-   * // Constrained to valid JSON
-   * ctx.initGrammar(grammar);
+   * // Constrained to valid JSON (handle-based API)
+   * const grammarHandle = ctx.createSampler(grammar);
+   * ctx.applySampler(grammarHandle, ctx.getLogits());
    * const token = ctx.sample({ temperature: 0.7 });
+   * ctx.acceptSamplerToken(grammarHandle, token);
    * ```
    */
   sample(params?: SamplingParams): number;
@@ -608,144 +610,6 @@ export interface SessionContext {
    */
   clearAndReseed(sinks: number[], tail: number[]): Promise<void>;
 
-  // ===== GRAMMAR-CONSTRAINED GENERATION =====
-
-  /**
-   * Initialize grammar parser (once per generation session)
-   *
-   * Grammars constrain generation to valid formats (JSON, XML, etc.).
-   * Parser tracks state across tokens to enforce rules.
-   *
-   * Call once before starting constrained generation.
-   * Use resetGrammar() to reuse same grammar for new generation.
-   *
-   * Cost: ~0.1-1ms depending on grammar complexity
-   *
-   * @param grammarStr GBNF grammar string (EBNF-like syntax)
-   * @example
-   * ```typescript
-   * // Force valid JSON
-   * const grammar = ctx.jsonSchemaToGrammar(JSON.stringify({
-   *   type: "object",
-   *   properties: {
-   *     name: { type: "string" },
-   *     age: { type: "number" }
-   *   }
-   * }));
-   *
-   * ctx.initGrammar(grammar);
-   *
-   * // Now sample() will only generate valid JSON
-   * const token = ctx.sample({ temperature: 0.7 });
-   * ```
-   */
-  initGrammar(grammarStr: string): void;
-
-  /**
-   * Apply grammar constraints to token scores (modifies in-place)
-   *
-   * Masks invalid tokens with -Infinity based on parser state.
-   * Call after getTokenScores(), before custom sampling.
-   *
-   * Flow: getTokenScores() → applyGrammar() → sample() → acceptToken()
-   *
-   * Thread safety: This method is synchronous and modifies the buffer
-   * in-place on the JS thread. Safe because it's called sequentially
-   * in the generation loop before any async operations.
-   *
-   * Cost: ~0.1-1ms depending on grammar complexity
-   *
-   * @param scoresBuffer Buffer from getTokenScores() (modified in-place)
-   * @throws Error if grammar not initialized (call initGrammar first)
-   * @example
-   * ```typescript
-   * // Custom sampling with grammar
-   * const buffer = ctx.getTokenScores();
-   * const scores = new Float32Array(buffer.buffer, buffer.byteOffset, buffer.length / 4);
-   *
-   * // Apply grammar constraints
-   * ctx.applyGrammar(buffer);
-   *
-   * // Now sample from constrained distribution
-   * const token = customSample(scores);
-   * ctx.acceptToken(token);
-   * ```
-   */
-  applyGrammar(scoresBuffer: Buffer): void;
-
-  /**
-   * Advance grammar parser with chosen token
-   *
-   * Updates parser state after sampling.
-   * MUST be called AFTER sampling, BEFORE next applyGrammar().
-   *
-   * This advances the stateful grammar parser through its rules.
-   * Without this, grammar constraints will be incorrect.
-   *
-   * Cost: <0.01ms
-   *
-   * @param tokenId Token that was sampled
-   * @example
-   * ```typescript
-   * const buffer = ctx.getTokenScores();
-   * ctx.applyGrammar(buffer);
-   *
-   * const scores = new Float32Array(buffer.buffer, buffer.byteOffset, buffer.length / 4);
-   * const token = customSample(scores);
-   *
-   * // MUST call acceptToken to advance parser
-   * ctx.acceptToken(token);
-   *
-   * // Now parser is ready for next token
-   * ```
-   */
-  acceptToken(tokenId: number): void;
-
-  /**
-   * Reset grammar parser to initial state
-   *
-   * Call at start of each new generation with same grammar.
-   * Parser returns to root state, ready to validate from beginning.
-   *
-   * Cost: <0.01ms
-   *
-   * @example
-   * ```typescript
-   * ctx.initGrammar(jsonGrammar);
-   *
-   * // First generation
-   * while (!done) {
-   *   const token = ctx.sample();
-   *   // ... generate ...
-   * }
-   *
-   * // Second generation - reuse same grammar
-   * ctx.resetGrammar();
-   * while (!done) {
-   *   const token = ctx.sample();
-   *   // ... generate ...
-   * }
-   * ```
-   */
-  resetGrammar(): void;
-
-  /**
-   * Free grammar resources
-   *
-   * Call when done with constrained generation.
-   * Releases parser memory.
-   *
-   * Cost: <0.01ms
-   *
-   * @example
-   * ```typescript
-   * ctx.initGrammar(grammar);
-   * // ... do constrained generation ...
-   * ctx.freeGrammar();
-   * ```
-   */
-  freeGrammar(): void;
-
   // ===== KV SEQUENCE OPERATIONS =====
 
   /**
@@ -817,9 +681,7 @@ export interface SessionContext {
    * Create a new grammar sampler (returns handle)
    *
    * Creates an independent grammar sampler instance with its own state.
-   *
-   * Unlike initGrammar() which uses a single internal sampler, this returns
-   * a handle that can be used with applySampler/acceptSamplerToken.
+   * Returns a handle that can be used with applySampler/acceptSamplerToken.
    * Multiple handles can coexist with independent parser states.
    *
    * Cost: ~0.1-1ms depending on grammar complexity
@@ -859,7 +721,6 @@ export interface SessionContext {
    * Accept token to advance grammar parser state (handle-based)
    *
    * Must be called after sampling to advance the grammar parser.
-   * This is the handle-based equivalent of acceptToken().
    *
    * @param handle Sampler handle from createSampler()
    * @param tokenId Token that was sampled
@@ -1186,7 +1047,7 @@ export interface SessionContext {
    * Convert JSON schema to GBNF grammar
    *
    * Generates grammar string for constrained JSON generation.
-   * Use with initGrammar() or sample({ grammar }).
+   * Use with createSampler() for grammar-constrained generation.
    *
    * Cost: ~1-10ms depending on schema complexity
    *
@@ -1204,7 +1065,7 @@ export interface SessionContext {
    * };
    *
    * const grammar = ctx.jsonSchemaToGrammar(JSON.stringify(schema));
-   * ctx.initGrammar(grammar);
+   * const handle = ctx.createSampler(grammar);
    * ```
    */
   jsonSchemaToGrammar(schemaJson: string): string;
@@ -1314,16 +1175,6 @@ export interface SessionContext {
 
   // ===== NATIVE REFERENCE IMPLEMENTATIONS =====
 
-  /**
-   * Compute entropy of current logits distribution
-   *
-   * Alternative entropy computation using native implementation.
-   * Equivalent to modelEntropy("nats") but may be faster.
-   *
-   * @returns Entropy in nats
-   */
-  computeEntropy(): number;
-
   /**
    * Sample greedily from current logits
    *
@@ -1360,6 +1211,44 @@ export interface SessionContext {
    * Context becomes unusable after disposal.
    */
   dispose(): void;
+
+  // ===== BRANCH API (internal, wrapped by Branch class) =====
+
+  /** @internal Create a new branch for parallel generation */
+  _branchCreate(seqId: number, position: number, params?: SamplingParams): number;
+
+  /** @internal Fork a branch to a new sequence */
+  _branchFork(handle: number, newSeqId: number): number;
+
+  /** @internal Capture logits into branch's snapshot */
+  _branchCaptureLogits(handle: number): void;
+
+  /** @internal Decode a single token and capture logits */
+  _branchDecodeAndCaptureOne(handle: number, token: number): void;
+
+  /** @internal Sample next token from branch's logits snapshot */
+  _branchSample(handle: number): number;
+
+  /** @internal Accept token (update sampler state for penalties) */
+  _branchAccept(handle: number, token: number): void;
+
+  /** @internal Get branch's sequence ID */
+  _branchGetSeqId(handle: number): number;
+
+  /** @internal Get branch's current position */
+  _branchGetPosition(handle: number): number;
+
+  /** @internal Get branch's perplexity */
+  _branchGetPerplexity(handle: number): number;
+
+  /** @internal Prune branch (remove KV cache entries and free handle) */
+  _branchPrune(handle: number): void;
+
+  /** @internal Destroy branch (free handle without removing KV cache) */
+  _branchDestroy(handle: number): void;
+
+  /** @internal Reseed branch sampler PRNG for diversity after fork */
+  _branchSamplerChainReseed(handle: number, seed: number): void;
 }
 
 /**
@@ -1502,3 +1391,140 @@ export function withLogits<T>(
   ctx: SessionContext,
   fn: (logits: Float32Array) => T
 ): T;
+
+/**
+ * Result from Branch.produce()
+ */
+export interface Produced {
+  /** Sampled token ID */
+  token: number;
+  /** Text representation of the token */
+  text: string;
+  /** Whether this is a stop token (EOS) */
+  isStop: boolean;
+}
+
+/**
+ * Forkable inference handle for covalent generation
+ *
+ * A Branch owns everything needed for independent generation: a KV cache
+ * sequence, sampler chain, logits snapshot, and perplexity tracker.
+ *
+ * Forking is cheap — the KV prefix is shared in memory (metadata-only operation under unified KV —
+ * no KV tensor buffers are copied), so sibling branches read from the same physical KV entries.
+ * Only tokens decoded after the fork point are exclusive to each branch.
+ *
+ * Branches form trees, not just flat lists. Fork from root for best-of-N,
+ * fork from children for MCTS/beam search, fork from a draft for speculative
+ * decoding.
+ *
+ * The produce/commit protocol separates sampling from state advancement:
+ * produce() samples without writing to KV, letting you inspect the result
+ * before deciding to commit().
+ *
+ * @example Best-of-N with perplexity selection
+ * ```typescript
+ * const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
+ * root.captureLogits();
+ *
+ * const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
+ *   const branch = root.fork(seqId);
+ *   branch.reseedSampler(1000 + i);
+ *   return branch;
+ * });
+ *
+ * for (let t = 0; t < 50; t++) {
+ *   for (const branch of candidates) {
+ *     const { token, isStop } = branch.produce();
+ *     if (isStop) continue;
+ *     branch.commit(token);
+ *   }
+ * }
+ *
+ * const best = candidates.reduce((a, b) => a.perplexity < b.perplexity ? a : b);
+ * for (const c of candidates) { if (c !== best) c.prune(); }
+ * ```
+ */
+export class Branch {
+  /**
+   * Create a root branch at the given position
+   *
+   * The branch takes ownership of the sequence and creates its own sampler
+   * chain from the provided params. Call captureLogits() after prefill to
+   * freeze the logit distribution before forking.
+   *
+   * @param ctx SessionContext to create branch on
+   * @param seqId Sequence ID for this branch
+   * @param position Starting position (typically prompt token count)
+   * @param params Sampling parameters (temperature, topP, etc.)
+   */
+  static create(
+    ctx: SessionContext,
+    seqId: number,
+    position: number,
+    params?: SamplingParams
+  ): Branch;
+
+  /**
+   * Fork this branch to a new sequence
+   *
+   * The child shares the parent's KV prefix in memory (metadata-only under unified KV, no KV buffer copy).
+   * Logits, sampler state, and perplexity tracker are cloned so the child
+   * can diverge independently. Fork from any branch — root or intermediate —
+   * to build arbitrarily deep trees.
+   *
+   * @param newSeqId Sequence ID for the forked branch
+   */
+  fork(newSeqId: number): Branch;
+
+  /** Freeze the current logit distribution into this branch. Essential before fork(). */
+  captureLogits(): void;
+
+  /** Decode a single token, write to KV, and capture resulting logits */
+  decodeAndCaptureOne(token: number): void;
+
+  /** Sample next token from branch's frozen logits snapshot */
+  sample(): number;
+
+  /** Accept token for repeat-penalty tracking */
+  accept(token: number): void;
+
+  /** Discard branch — remove its divergent KV entries and free the handle (use for losers) */
+  prune(): void;
+
+  /** Release handle but keep KV entries intact (use for winners, continue with raw ops) */
+  destroy(): void;
+
+  /**
+   * Reseed the sampler's PRNG for diversity after fork()
+   *
+   * CRITICAL for parallel generation: Without reseeding, all forked branches
+   * produce identical outputs because they share the same PRNG state.
+   *
+   * Only affects stochastic samplers (temperature > 0). Greedy samplers are unchanged.
+   *
+   * @param seed - New seed for the PRNG
+   */
+  reseedSampler(seed: number): void;
+
+  /** Sample next token without advancing state. Inspect before committing. */
+  produce(): Produced;
+
+  /** Accept and advance — write token to KV and update branch state. */
+  commit(token: number): void;
+
+  /** Branch's sequence ID */
+  readonly seqId: number;
+
+  /** Branch's current position */
+  readonly position: number;
+
+  /** Branch's perplexity */
+  readonly perplexity: number;
+
+  /** Internal handle (for debugging) */
+  readonly handle: number;
+
+  /** Whether this branch has been disposed */
+  readonly disposed: boolean;
+}

package/lib/index.js

@@ -23,11 +23,11 @@
  * // Safe logits access (Runtime Borrow Checker pattern)
  * const entropy = withLogits(ctx, (logits) => {
  *   // logits is valid here - use synchronously only!
- *   return computeEntropy(logits);
+ *   return myComputeEntropy(logits);
  * });
  *
  * // Or with native reference implementations (for testing)
- * const nativeEntropy = ctx.computeEntropy();
+ * const entropy = ctx.modelEntropy();
  * const token = ctx.greedySample();
  *
  * // Cleanup
@@ -203,7 +203,14 @@ function withLogits(ctx, fn) {
   return result;
 }
 
+const { Branch } = require('./Branch');
+
 module.exports = {
+  /**
+   * Branch class for parallel generation
+   * @see Branch.create()
+   */
+  Branch,
   /**
    * Create a new inference context
    *

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lloyal-labs/lloyal.node",
-  "version": "1.0.6-alpha",
+  "version": "1.0.8",
   "description": "Node.js client for liblloyal+llama.cpp",
   "main": "lib/index.js",
   "types": "lib/index.d.ts",
@@ -19,6 +19,8 @@
     "test": "npm run test:api && npm run test:e2e",
     "test:api": "node test/api.js",
     "test:e2e": "node test/e2e.js",
+    "test:examples": "node test/examples.js",
+    "sync:llama-cpp": "node scripts/sync-llama-cpp.js",
     "example": "node examples/chat/chat.mjs"
   },
   "repository": {
@@ -48,22 +50,23 @@
   "devDependencies": {
     "cmake-js": "^7.4.0",
     "glob": "^11.0.0",
-    "typedoc": "^0.27.5"
+    "typedoc": "^0.28.16",
+    "typedoc-rhineai-theme": "^1.2.0"
   },
   "optionalDependencies": {
-    "@lloyal-labs/lloyal.node-darwin-arm64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-darwin-x64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-arm64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-arm64-cuda": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-arm64-vulkan": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-x64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-x64-cuda": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-x64-vulkan": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-arm64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-arm64-vulkan": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-x64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-x64-cuda": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-x64-vulkan": "1.0.6-alpha"
+    "@lloyal-labs/lloyal.node-darwin-arm64": "1.0.8",
+    "@lloyal-labs/lloyal.node-darwin-x64": "1.0.8",
+    "@lloyal-labs/lloyal.node-linux-arm64": "1.0.8",
+    "@lloyal-labs/lloyal.node-linux-arm64-cuda": "1.0.8",
+    "@lloyal-labs/lloyal.node-linux-arm64-vulkan": "1.0.8",
+    "@lloyal-labs/lloyal.node-linux-x64": "1.0.8",
+    "@lloyal-labs/lloyal.node-linux-x64-cuda": "1.0.8",
+    "@lloyal-labs/lloyal.node-linux-x64-vulkan": "1.0.8",
+    "@lloyal-labs/lloyal.node-win32-arm64": "1.0.8",
+    "@lloyal-labs/lloyal.node-win32-arm64-vulkan": "1.0.8",
+    "@lloyal-labs/lloyal.node-win32-x64": "1.0.8",
+    "@lloyal-labs/lloyal.node-win32-x64-cuda": "1.0.8",
+    "@lloyal-labs/lloyal.node-win32-x64-vulkan": "1.0.8"
   },
   "engines": {
     "node": ">=22.0.0"

package/scripts/create-platform-package.js

@@ -79,8 +79,9 @@ if (osName === 'darwin') {
   });
 
 } else if (osName === 'linux') {
-  // Copy all .so files
-  const sos = fs.readdirSync(BUILD_DIR).filter(f => f.endsWith('.so'));
+  // Copy all .so files including versioned variants (e.g., libllama.so.0, libllama.so.0.0.X)
+  // llama.cpp sets SOVERSION, producing versioned names that the binary references at runtime
+  const sos = fs.readdirSync(BUILD_DIR).filter(f => /\.so(\.\d+)*$/.test(f));
   if (sos.length > 0) {
     sos.forEach(so => {
       fs.copyFileSync(path.join(BUILD_DIR, so), path.join(BIN_DIR, so));

package/scripts/download-test-models.sh

@@ -26,5 +26,15 @@ else
   echo "  ✓ nomic-embed-text already exists"
 fi
 
+# slim-summary-tool (1.7GB) - Summary sidecar for dynamic sinks
+if [ ! -f "slim-summarize.gguf" ]; then
+  echo "  → Downloading slim-summarize.gguf..."
+  curl -L -o "slim-summarize.gguf" \
+    "https://huggingface.co/llmware/slim-summary-tool/resolve/main/slim-summary-tool.gguf"
+  echo "  ✓ Downloaded slim-summarize"
+else
+  echo "  ✓ slim-summarize already exists"
+fi
+
 echo ""
 echo "✅ All test models ready"

package/scripts/sync-llama-cpp.js

@@ -0,0 +1,117 @@
+#!/usr/bin/env node
+/**
+ * Sync llama.cpp submodule to match liblloyal's .llama-cpp-version
+ *
+ * Single source of truth: liblloyal/.llama-cpp-version contains the tag
+ * that the llama.cpp submodule should be checked out at.
+ *
+ * Usage:
+ *   node scripts/sync-llama-cpp.js          # Sync submodule to target tag
+ *   node scripts/sync-llama-cpp.js --check  # Validate match (CI mode)
+ */
+
+const { execSync } = require('child_process');
+const fs = require('fs');
+const path = require('path');
+
+const ROOT = path.join(__dirname, '..');
+const VERSION_FILE = path.join(ROOT, 'liblloyal', '.llama-cpp-version');
+const LLAMA_CPP_DIR = path.join(ROOT, 'llama.cpp');
+
+const CHECK_ONLY = process.argv.includes('--check');
+
+// --- Read target version ---
+
+if (!fs.existsSync(VERSION_FILE)) {
+  console.error('[sync-llama-cpp] Error: liblloyal/.llama-cpp-version not found.');
+  console.error('[sync-llama-cpp] Make sure liblloyal submodule is initialized:');
+  console.error('[sync-llama-cpp]   git submodule update --init --recursive');
+  process.exit(1);
+}
+
+const versionFileContent = fs.readFileSync(VERSION_FILE, 'utf8');
+const targetVersion = versionFileContent
+  .split('\n')
+  .filter(line => !line.startsWith('#') && line.trim().length > 0)
+  [0]
+  ?.trim();
+
+if (!targetVersion) {
+  console.error('[sync-llama-cpp] Error: Could not parse version from liblloyal/.llama-cpp-version');
+  process.exit(1);
+}
+
+console.log(`[sync-llama-cpp] Target llama.cpp version: ${targetVersion}`);
+
+// --- Check llama.cpp submodule exists ---
+
+if (!fs.existsSync(path.join(LLAMA_CPP_DIR, '.git'))) {
+  console.error('[sync-llama-cpp] Error: llama.cpp submodule not initialized.');
+  console.error('[sync-llama-cpp] Run: git submodule update --init --recursive');
+  process.exit(1);
+}
+
+// --- Helper ---
+
+function exec(cmd, opts = {}) {
+  return execSync(cmd, { cwd: LLAMA_CPP_DIR, encoding: 'utf8', stdio: 'pipe', ...opts }).trim();
+}
+
+// --- Get current llama.cpp state ---
+
+const currentSha = exec('git rev-parse HEAD');
+
+// Resolve target tag to SHA (may need to fetch in shallow clones)
+let targetSha;
+try {
+  targetSha = exec(`git rev-parse ${targetVersion}`);
+} catch {
+  // Tag not available locally — fetch it
+  console.log(`[sync-llama-cpp] Tag ${targetVersion} not found locally, fetching...`);
+  try {
+    exec(`git fetch origin tag ${targetVersion} --no-tags --depth 1`);
+    targetSha = exec(`git rev-parse ${targetVersion}`);
+  } catch (e) {
+    console.error(`[sync-llama-cpp] Error: Tag ${targetVersion} not found in remote.`);
+    console.error(`[sync-llama-cpp] Verify tag exists: https://github.com/ggml-org/llama.cpp/releases/tag/${targetVersion}`);
+    process.exit(1);
+  }
+}
+
+const currentShort = currentSha.slice(0, 7);
+const targetShort = targetSha.slice(0, 7);
+
+console.log(`[sync-llama-cpp] Current: ${currentShort} (${currentSha})`);
+console.log(`[sync-llama-cpp] Target:  ${targetShort} (${targetVersion})`);
+
+if (currentSha === targetSha) {
+  console.log(`[sync-llama-cpp] llama.cpp submodule matches ${targetVersion}.`);
+  process.exit(0);
+}
+
+// --- Mismatch ---
+
+if (CHECK_ONLY) {
+  console.error(`\n[sync-llama-cpp] MISMATCH: llama.cpp submodule is at ${currentShort}, expected ${targetVersion} (${targetShort})`);
+  console.error(`[sync-llama-cpp] Fix: npm run sync:llama-cpp`);
+  process.exit(1);
+}
+
+// --- Sync ---
+
+console.log(`[sync-llama-cpp] Checking out ${targetVersion}...`);
+
+try {
+  exec(`git checkout ${targetVersion}`);
+} catch {
+  exec(`git fetch origin tag ${targetVersion} --no-tags --depth 1`);
+  exec(`git checkout ${targetVersion}`);
+}
+
+const newShort = exec('git rev-parse --short HEAD');
+console.log(`[sync-llama-cpp] llama.cpp now at: ${newShort} (${targetVersion})`);
+console.log('');
+console.log('[sync-llama-cpp] Next steps:');
+console.log('  1. Build and test:  npm run build && npm test');
+console.log('  2. Stage changes:   git add llama.cpp');
+console.log('  3. Commit:          git commit -m "chore(deps): sync llama.cpp to ' + targetVersion + '"');