@lloyal-labs/lloyal.node 1.0.6-alpha → 1.0.7

package/README.md

@@ -2,7 +2,7 @@
 
 **Advanced edge inference for Node.js**
 
-Inference with forkable state — KV cache, grammar, metrics all clone atomically. Entropy and surprisal mid-generation. Multi-sequence parallel exploration. The control surface llama.cpp exposes, in TypeScript.
+A llama.cpp control surface in TypeScript with atomic inference state forking. Real time rolling perplexity/entropy/surprisal and multi-sequence parallel exploration primitives.
 
 ```bash
 npm install @lloyal-labs/lloyal.node
@@ -29,8 +29,8 @@ Working examples demonstrate each capability:
 
 | Example                                   | What It Demonstrates                                                          |
 | ----------------------------------------- | ----------------------------------------------------------------------------- |
-| [`best-of-n/`](./examples/best-of-n/)     | Multi-sequence generation, PPL selection, captured logits for fair comparison |
-| [`speculative/`](./examples/speculative/) | KV forking, draft/verify/accept/reject, `kvCacheRemove` for rejected tokens   |
+| [`best-of-n/`](./examples/best-of-n/)     | Branch API parallel generation, PPL selection, fork/produce/commit            |
+| [`speculative/`](./examples/speculative/) | Branch API fork/prune, draft/verify/accept/reject, bonus token sampling       |
 | [`entropy/`](./examples/entropy/)         | Entropy Decision Tree — `modelEntropy()` mid-generation as control signal     |
 | [`grammar/`](./examples/grammar/)         | Pull loop with generators, JSON schema constraints, KV + grammar branching    |
 | [`streaming/`](./examples/streaming/)     | Infinite context via BlinkKV, `clearAndReseed`, perplexity tracking           |
@@ -50,49 +50,64 @@ Each example has a README explaining the pattern in depth.
 
 ## Core Patterns
 
-### Forkable State
+### Branch API
 
-KV cache, grammar parser, and perplexity trackers all live behind handles. Handles clone atomically.
+`Branch` is the primary API for parallel generation. Each branch owns a KV cache sequence, sampler chain, logits snapshot, and perplexity tracker. Fork a branch to explore alternatives, compare by perplexity, prune losers.
 
-**Two forking strategies:**
+```javascript
+import { createContext, Branch } from '@lloyal-labs/lloyal.node';
 
-| Approach             | Method                            | Use Case                                     |
-| -------------------- | --------------------------------- | -------------------------------------------- |
-| **Tag copy**         | `kvSeqCopy(src, dst)`             | Parallel branches with different seqIds      |
-| **Snapshot/restore** | `kvCacheSave()` / `kvCacheLoad()` | Sequential exploration, return to checkpoint |
+const ctx = await createContext({ modelPath: './model.gguf', nSeqMax: 8 });
+const tokens = await ctx.tokenize('Once upon a time');
+await ctx.decode(tokens, 0, 0);
 
-[`examples/best-of-n/`](./examples/best-of-n/) uses tag copy — each candidate gets its own seqId, branches run in parallel:
+// Create root branch, capture logits from prefill
+const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
+root.captureLogits();
 
-```javascript
-ctx.kvSeqCopy(0, seqId); // O(1) tag copy, branch diverges on seqId
-```
+// Fork N candidates — each gets copied KV, logits, sampler, perplexity
+const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
+  const branch = root.fork(seqId);
+  branch.reseedSampler(1000 + i); // Unique PRNG per branch
+  return branch;
+});
 
-[`examples/grammar/`](./examples/grammar/) uses snapshot/restore — save state, explore branches sequentially, restore between each:
+// Generate in parallel (interleaved round-robin)
+for (let t = 0; t < 50; t++) {
+  for (const branch of candidates) {
+    const { token, isStop } = branch.produce(); // Sample (no KV write)
+    if (isStop) continue;
+    branch.commit(token); // Accept + decode + capture
+  }
+}
 
-```javascript
-const snapshot = await ctx.kvCacheSave(0); // Save checkpoint
-// ... explore branch ...
-await ctx.kvCacheLoad(0, snapshot); // Return to checkpoint
+// Select best by perplexity, prune losers
+const best = candidates.reduce((a, b) => a.perplexity < b.perplexity ? a : b);
+for (const c of candidates) { if (c !== best) c.prune(); }
 ```
 
-Both approaches also fork grammar state with `cloneSampler()` when grammar constraints are involved.
+**What `fork()` clones:** KV cache sequence, logits snapshot, sampler chain (penalties + PRNG), perplexity tracker. Under unified KV (the default), forking is a metadata-only operation — no KV tensor buffers are copied.
 
-### Captured Logits
+**Use cases:** Best-of-N sampling, speculative decoding, MCTS/LATS tree search, beam search.
 
-After decode, logits represent P(next_token | context). When forking to multiple sequences, capture logits for fair comparison:
+See [`examples/best-of-n/`](./examples/best-of-n/) and [`examples/speculative/`](./examples/speculative/) for complete patterns.
 
-```javascript
-// Capture after prefill
-const capturedLogits = new Float32Array(ctx.getLogits());
+### Low-Level Forking
 
-// All candidates sample first token from same distribution
-const token = sampleWithStrategy(capturedLogits, { params, workspace, prng });
+For fine-grained control without the Branch wrapper, raw KV and state operations are available:
 
-// Compute surprisal from captured logits (native C++)
-const surprisal = ctx.modelSurprisal(token, 'nats', capturedLogits);
-```
+| Approach             | Method                            | Use Case                                     |
+| -------------------- | --------------------------------- | -------------------------------------------- |
+| **Tag copy**         | `kvSeqCopy(src, dst)`             | Parallel branches with different seqIds      |
+| **Snapshot/restore** | `kvCacheSave()` / `kvCacheLoad()` | Sequential exploration, return to checkpoint |
 
-See [`examples/best-of-n/`](./examples/best-of-n/) for the full pattern.
+[`examples/grammar/`](./examples/grammar/) uses snapshot/restore — save state, explore branches sequentially, restore between each:
+
+```javascript
+const snapshot = await ctx.kvCacheSave(0); // Save checkpoint
+// ... explore branch ...
+await ctx.kvCacheLoad(0, snapshot); // Return to checkpoint
+```
 
 ### Entropy as Control Signal
 

package/lib/Branch.js

@@ -0,0 +1,268 @@
+/**
+ * Branch - Forkable inference handle for covalent generation
+ *
+ * A Branch owns everything needed for independent generation: a KV cache
+ * sequence, sampler chain, logits snapshot, and perplexity tracker.
+ *
+ * Forking is cheap — the KV prefix is shared in memory (metadata-only operation under unified KV —
+ * no KV tensor buffers are copied), so sibling branches read from the same physical KV entries.
+ * Only tokens decoded after the fork point are exclusive to each branch.
+ * This is the covalent property: branches share a bond (common prefix)
+ * while diverging independently.
+ *
+ * Branches form trees, not just flat lists. Fork from root for best-of-N,
+ * fork from children for MCTS/beam search, fork from a draft for speculative
+ * decoding.
+ *
+ * The produce/commit protocol separates sampling from state advancement:
+ * produce() samples without writing to KV, letting you inspect the result
+ * before deciding to commit(). This two-phase split is what makes speculative
+ * verification and tree search natural.
+ *
+ * @example Best-of-N with perplexity selection
+ * ```js
+ * const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
+ * root.captureLogits();
+ *
+ * const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
+ *   const branch = root.fork(seqId);
+ *   branch.reseedSampler(1000 + i);
+ *   return branch;
+ * });
+ *
+ * for (let t = 0; t < 50; t++) {
+ *   for (const branch of candidates) {
+ *     const { token, isStop } = branch.produce();
+ *     if (isStop) continue;
+ *     branch.commit(token);
+ *   }
+ * }
+ *
+ * const best = candidates.reduce((a, b) => a.perplexity < b.perplexity ? a : b);
+ * for (const c of candidates) { if (c !== best) c.prune(); }
+ * ```
+ */
+
+class Branch {
+  /**
+   * @param {SessionContext} ctx
+   * @param {number} handle
+   */
+  constructor(ctx, handle) {
+    this._ctx = ctx;
+    this._handle = handle;
+    this._disposed = false;
+  }
+
+  /**
+   * Create a root branch at the given position
+   *
+   * The branch takes ownership of the sequence and creates its own sampler
+   * chain from the provided params. Call captureLogits() after prefill to
+   * freeze the logit distribution before forking.
+   *
+   * @param {SessionContext} ctx - SessionContext to create branch on
+   * @param {number} seqId - Sequence ID for this branch
+   * @param {number} position - Starting position (typically prompt token count)
+   * @param {SamplingParams} [params] - Sampling parameters (temperature, topP, etc.)
+   * @returns {Branch} New Branch instance
+   */
+  static create(ctx, seqId, position, params) {
+    const handle = ctx._branchCreate(seqId, position, params);
+    return new Branch(ctx, handle);
+  }
+
+  /**
+   * Fork this branch to a new sequence
+   *
+   * The child shares the parent's KV prefix in memory (metadata-only under unified KV, no KV buffer copy).
+   * Logits, sampler state, and perplexity tracker are cloned so the child
+   * can diverge independently. Fork from any branch — root or intermediate —
+   * to build arbitrarily deep trees.
+   *
+   * Call reseedSampler() on each child for stochastic diversity.
+   *
+   * @param {number} newSeqId - Sequence ID for the forked branch
+   * @returns {Branch} New forked Branch
+   */
+  fork(newSeqId) {
+    this._ensureNotDisposed();
+    const newHandle = this._ctx._branchFork(this._handle, newSeqId);
+    return new Branch(this._ctx, newHandle);
+  }
+
+  /**
+   * Freeze the current logit distribution into this branch
+   *
+   * Logits are ephemeral — they're overwritten on the next decode() call.
+   * Capturing preserves them so this branch (and any forks from it) can
+   * sample from the same distribution. Essential before fork().
+   */
+  captureLogits() {
+    this._ensureNotDisposed();
+    this._ctx._branchCaptureLogits(this._handle);
+  }
+
+  /**
+   * Single-token forward pass with logit snapshot
+   *
+   * Runs one decode step (writing the token's KV entries), advances position,
+   * and captures the resulting logits for the next sample() call.
+   *
+   * @param {number} token - Token to decode
+   */
+  decodeAndCaptureOne(token) {
+    this._ensureNotDisposed();
+    this._ctx._branchDecodeAndCaptureOne(this._handle, token);
+  }
+
+  /**
+   * Sample next token from branch's logits snapshot
+   *
+   * Applies the branch's full sampler chain (top-k, top-p, temperature,
+   * repeat/presence penalties) to the captured logits.
+   *
+   * @returns {number} Sampled token ID
+   */
+  sample() {
+    this._ensureNotDisposed();
+    return this._ctx._branchSample(this._handle);
+  }
+
+  /**
+   * Record token in the sampler's repeat/presence penalty window
+   *
+   * @param {number} token - Token to accept
+   */
+  accept(token) {
+    this._ensureNotDisposed();
+    this._ctx._branchAccept(this._handle, token);
+  }
+
+  /**
+   * Discard this branch entirely — remove its KV entries and free the handle
+   *
+   * Use for losers: branches whose generation you want to erase completely.
+   * Only removes KV entries divergent from the shared prefix; sibling
+   * branches are unaffected.
+   */
+  prune() {
+    if (this._disposed) return;
+    this._ctx._branchPrune(this._handle);
+    this._disposed = true;
+  }
+
+  /**
+   * Release the handle but keep KV cache entries intact
+   *
+   * Use for winners: you're done branching but want to continue generation
+   * on this sequence using raw ctx.decode()/ctx.sample() calls. The KV
+   * cache entries remain at their current positions.
+   */
+  destroy() {
+    if (this._disposed) return;
+    this._ctx._branchDestroy(this._handle);
+    this._disposed = true;
+  }
+
+  /**
+   * Reseed the sampler's PRNG for diversity after fork()
+   *
+   * CRITICAL for parallel generation: Without reseeding, all forked branches
+   * produce identical outputs because they share the same PRNG state.
+   *
+   * Only affects stochastic samplers (temperature > 0). Greedy samplers are unchanged.
+   *
+   * @param {number} seed - New seed for the PRNG
+   *
+   * @example
+   * ```js
+   * const root = Branch.create(ctx, 0, pos, { temperature: 0.9 });
+   * root.captureLogits();
+   *
+   * // Fork and reseed for diversity
+   * const branches = [1, 2, 3, 4, 5].map((seqId, i) => {
+   *   const branch = root.fork(seqId);
+   *   branch.reseedSampler(1000 + i);  // Each branch gets unique seed
+   *   return branch;
+   * });
+   * ```
+   */
+  reseedSampler(seed) {
+    this._ensureNotDisposed();
+    this._ctx._branchSamplerChainReseed(this._handle, seed);
+  }
+
+  /**
+   * Sample the next token without advancing state
+   *
+   * No KV write, no position update. Inspect the result before deciding
+   * to commit() — this separation is what enables speculative verification
+   * and conditional branching.
+   *
+   * @returns {{ token: number, text: string, isStop: boolean }}
+   */
+  produce() {
+    this._ensureNotDisposed();
+    const token = this.sample();
+    return {
+      token,
+      text: this._ctx.tokenToText(token),
+      isStop: this._ctx.isStopToken(token),
+    };
+  }
+
+  /**
+   * Accept and advance — write token to KV and update branch state
+   *
+   * Accepts the token for repeat-penalty tracking, decodes it (writing to
+   * KV cache), and captures the resulting logits for the next produce() call.
+   *
+   * @param {number} token - Token to commit (from produce())
+   */
+  commit(token) {
+    this._ensureNotDisposed();
+    this.accept(token);
+    this.decodeAndCaptureOne(token);
+  }
+
+  // ===== ACCESSORS =====
+
+  /** @returns {number} Branch's sequence ID */
+  get seqId() {
+    this._ensureNotDisposed();
+    return this._ctx._branchGetSeqId(this._handle);
+  }
+
+  /** @returns {number} Branch's current position (number of tokens decoded) */
+  get position() {
+    this._ensureNotDisposed();
+    return this._ctx._branchGetPosition(this._handle);
+  }
+
+  /** @returns {number} Branch's perplexity (exp of mean surprisal) */
+  get perplexity() {
+    this._ensureNotDisposed();
+    return this._ctx._branchGetPerplexity(this._handle);
+  }
+
+  /** @returns {number} Internal handle (for debugging) */
+  get handle() {
+    return this._handle;
+  }
+
+  /** @returns {boolean} Whether this branch has been disposed */
+  get disposed() {
+    return this._disposed;
+  }
+
+  // ===== INTERNAL =====
+
+  _ensureNotDisposed() {
+    if (this._disposed) {
+      throw new Error('Branch has been disposed');
+    }
+  }
+}
+
+module.exports = { Branch };

package/lib/index.d.ts

@@ -347,9 +347,11 @@ export interface SessionContext {
    * // Creative generation
    * const token = ctx.sample({ temperature: 0.9 });
    *
-   * // Constrained to valid JSON
-   * ctx.initGrammar(grammar);
+   * // Constrained to valid JSON (handle-based API)
+   * const grammarHandle = ctx.createSampler(grammar);
+   * ctx.applySampler(grammarHandle, ctx.getLogits());
    * const token = ctx.sample({ temperature: 0.7 });
+   * ctx.acceptSamplerToken(grammarHandle, token);
    * ```
    */
   sample(params?: SamplingParams): number;
@@ -608,144 +610,6 @@ export interface SessionContext {
    */
   clearAndReseed(sinks: number[], tail: number[]): Promise<void>;
 
-  // ===== GRAMMAR-CONSTRAINED GENERATION =====
-
-  /**
-   * Initialize grammar parser (once per generation session)
-   *
-   * Grammars constrain generation to valid formats (JSON, XML, etc.).
-   * Parser tracks state across tokens to enforce rules.
-   *
-   * Call once before starting constrained generation.
-   * Use resetGrammar() to reuse same grammar for new generation.
-   *
-   * Cost: ~0.1-1ms depending on grammar complexity
-   *
-   * @param grammarStr GBNF grammar string (EBNF-like syntax)
-   * @example
-   * ```typescript
-   * // Force valid JSON
-   * const grammar = ctx.jsonSchemaToGrammar(JSON.stringify({
-   *   type: "object",
-   *   properties: {
-   *     name: { type: "string" },
-   *     age: { type: "number" }
-   *   }
-   * }));
-   *
-   * ctx.initGrammar(grammar);
-   *
-   * // Now sample() will only generate valid JSON
-   * const token = ctx.sample({ temperature: 0.7 });
-   * ```
-   */
-  initGrammar(grammarStr: string): void;
-
-  /**
-   * Apply grammar constraints to token scores (modifies in-place)
-   *
-   * Masks invalid tokens with -Infinity based on parser state.
-   * Call after getTokenScores(), before custom sampling.
-   *
-   * Flow: getTokenScores() → applyGrammar() → sample() → acceptToken()
-   *
-   * Thread safety: This method is synchronous and modifies the buffer
-   * in-place on the JS thread. Safe because it's called sequentially
-   * in the generation loop before any async operations.
-   *
-   * Cost: ~0.1-1ms depending on grammar complexity
-   *
-   * @param scoresBuffer Buffer from getTokenScores() (modified in-place)
-   * @throws Error if grammar not initialized (call initGrammar first)
-   * @example
-   * ```typescript
-   * // Custom sampling with grammar
-   * const buffer = ctx.getTokenScores();
-   * const scores = new Float32Array(buffer.buffer, buffer.byteOffset, buffer.length / 4);
-   *
-   * // Apply grammar constraints
-   * ctx.applyGrammar(buffer);
-   *
-   * // Now sample from constrained distribution
-   * const token = customSample(scores);
-   * ctx.acceptToken(token);
-   * ```
-   */
-  applyGrammar(scoresBuffer: Buffer): void;
-
-  /**
-   * Advance grammar parser with chosen token
-   *
-   * Updates parser state after sampling.
-   * MUST be called AFTER sampling, BEFORE next applyGrammar().
-   *
-   * This advances the stateful grammar parser through its rules.
-   * Without this, grammar constraints will be incorrect.
-   *
-   * Cost: <0.01ms
-   *
-   * @param tokenId Token that was sampled
-   * @example
-   * ```typescript
-   * const buffer = ctx.getTokenScores();
-   * ctx.applyGrammar(buffer);
-   *
-   * const scores = new Float32Array(buffer.buffer, buffer.byteOffset, buffer.length / 4);
-   * const token = customSample(scores);
-   *
-   * // MUST call acceptToken to advance parser
-   * ctx.acceptToken(token);
-   *
-   * // Now parser is ready for next token
-   * ```
-   */
-  acceptToken(tokenId: number): void;
-
-  /**
-   * Reset grammar parser to initial state
-   *
-   * Call at start of each new generation with same grammar.
-   * Parser returns to root state, ready to validate from beginning.
-   *
-   * Cost: <0.01ms
-   *
-   * @example
-   * ```typescript
-   * ctx.initGrammar(jsonGrammar);
-   *
-   * // First generation
-   * while (!done) {
-   *   const token = ctx.sample();
-   *   // ... generate ...
-   * }
-   *
-   * // Second generation - reuse same grammar
-   * ctx.resetGrammar();
-   * while (!done) {
-   *   const token = ctx.sample();
-   *   // ... generate ...
-   * }
-   * ```
-   */
-  resetGrammar(): void;
-
-  /**
-   * Free grammar resources
-   *
-   * Call when done with constrained generation.
-   * Releases parser memory.
-   *
-   * Cost: <0.01ms
-   *
-   * @example
-   * ```typescript
-   * ctx.initGrammar(grammar);
-   * // ... do constrained generation ...
-   * ctx.freeGrammar();
-   * ```
-   */
-  freeGrammar(): void;
-
   // ===== KV SEQUENCE OPERATIONS =====
 
   /**
@@ -817,9 +681,7 @@ export interface SessionContext {
    * Create a new grammar sampler (returns handle)
    *
    * Creates an independent grammar sampler instance with its own state.
-   *
-   * Unlike initGrammar() which uses a single internal sampler, this returns
-   * a handle that can be used with applySampler/acceptSamplerToken.
+   * Returns a handle that can be used with applySampler/acceptSamplerToken.
    * Multiple handles can coexist with independent parser states.
    *
    * Cost: ~0.1-1ms depending on grammar complexity
@@ -859,7 +721,6 @@ export interface SessionContext {
    * Accept token to advance grammar parser state (handle-based)
    *
    * Must be called after sampling to advance the grammar parser.
-   * This is the handle-based equivalent of acceptToken().
    *
    * @param handle Sampler handle from createSampler()
    * @param tokenId Token that was sampled
@@ -1186,7 +1047,7 @@ export interface SessionContext {
    * Convert JSON schema to GBNF grammar
    *
    * Generates grammar string for constrained JSON generation.
-   * Use with initGrammar() or sample({ grammar }).
+   * Use with createSampler() for grammar-constrained generation.
    *
    * Cost: ~1-10ms depending on schema complexity
    *
@@ -1204,7 +1065,7 @@ export interface SessionContext {
    * };
    *
    * const grammar = ctx.jsonSchemaToGrammar(JSON.stringify(schema));
-   * ctx.initGrammar(grammar);
+   * const handle = ctx.createSampler(grammar);
    * ```
    */
   jsonSchemaToGrammar(schemaJson: string): string;
@@ -1314,16 +1175,6 @@ export interface SessionContext {
 
   // ===== NATIVE REFERENCE IMPLEMENTATIONS =====
 
-  /**
-   * Compute entropy of current logits distribution
-   *
-   * Alternative entropy computation using native implementation.
-   * Equivalent to modelEntropy("nats") but may be faster.
-   *
-   * @returns Entropy in nats
-   */
-  computeEntropy(): number;
-
   /**
    * Sample greedily from current logits
    *
@@ -1360,6 +1211,44 @@ export interface SessionContext {
    * Context becomes unusable after disposal.
    */
   dispose(): void;
+
+  // ===== BRANCH API (internal, wrapped by Branch class) =====
+
+  /** @internal Create a new branch for parallel generation */
+  _branchCreate(seqId: number, position: number, params?: SamplingParams): number;
+
+  /** @internal Fork a branch to a new sequence */
+  _branchFork(handle: number, newSeqId: number): number;
+
+  /** @internal Capture logits into branch's snapshot */
+  _branchCaptureLogits(handle: number): void;
+
+  /** @internal Decode a single token and capture logits */
+  _branchDecodeAndCaptureOne(handle: number, token: number): void;
+
+  /** @internal Sample next token from branch's logits snapshot */
+  _branchSample(handle: number): number;
+
+  /** @internal Accept token (update sampler state for penalties) */
+  _branchAccept(handle: number, token: number): void;
+
+  /** @internal Get branch's sequence ID */
+  _branchGetSeqId(handle: number): number;
+
+  /** @internal Get branch's current position */
+  _branchGetPosition(handle: number): number;
+
+  /** @internal Get branch's perplexity */
+  _branchGetPerplexity(handle: number): number;
+
+  /** @internal Prune branch (remove KV cache entries and free handle) */
+  _branchPrune(handle: number): void;
+
+  /** @internal Destroy branch (free handle without removing KV cache) */
+  _branchDestroy(handle: number): void;
+
+  /** @internal Reseed branch sampler PRNG for diversity after fork */
+  _branchSamplerChainReseed(handle: number, seed: number): void;
 }
 
 /**
@@ -1502,3 +1391,140 @@ export function withLogits<T>(
   ctx: SessionContext,
   fn: (logits: Float32Array) => T
 ): T;
+
+/**
+ * Result from Branch.produce()
+ */
+export interface Produced {
+  /** Sampled token ID */
+  token: number;
+  /** Text representation of the token */
+  text: string;
+  /** Whether this is a stop token (EOS) */
+  isStop: boolean;
+}
+
+/**
+ * Forkable inference handle for covalent generation
+ *
+ * A Branch owns everything needed for independent generation: a KV cache
+ * sequence, sampler chain, logits snapshot, and perplexity tracker.
+ *
+ * Forking is cheap — the KV prefix is shared in memory (metadata-only operation under unified KV —
+ * no KV tensor buffers are copied), so sibling branches read from the same physical KV entries.
+ * Only tokens decoded after the fork point are exclusive to each branch.
+ *
+ * Branches form trees, not just flat lists. Fork from root for best-of-N,
+ * fork from children for MCTS/beam search, fork from a draft for speculative
+ * decoding.
+ *
+ * The produce/commit protocol separates sampling from state advancement:
+ * produce() samples without writing to KV, letting you inspect the result
+ * before deciding to commit().
+ *
+ * @example Best-of-N with perplexity selection
+ * ```typescript
+ * const root = Branch.create(ctx, 0, tokens.length, { temperature: 0.8 });
+ * root.captureLogits();
+ *
+ * const candidates = [1, 2, 3, 4, 5].map((seqId, i) => {
+ *   const branch = root.fork(seqId);
+ *   branch.reseedSampler(1000 + i);
+ *   return branch;
+ * });
+ *
+ * for (let t = 0; t < 50; t++) {
+ *   for (const branch of candidates) {
+ *     const { token, isStop } = branch.produce();
+ *     if (isStop) continue;
+ *     branch.commit(token);
+ *   }
+ * }
+ *
+ * const best = candidates.reduce((a, b) => a.perplexity < b.perplexity ? a : b);
+ * for (const c of candidates) { if (c !== best) c.prune(); }
+ * ```
+ */
+export class Branch {
+  /**
+   * Create a root branch at the given position
+   *
+   * The branch takes ownership of the sequence and creates its own sampler
+   * chain from the provided params. Call captureLogits() after prefill to
+   * freeze the logit distribution before forking.
+   *
+   * @param ctx SessionContext to create branch on
+   * @param seqId Sequence ID for this branch
+   * @param position Starting position (typically prompt token count)
+   * @param params Sampling parameters (temperature, topP, etc.)
+   */
+  static create(
+    ctx: SessionContext,
+    seqId: number,
+    position: number,
+    params?: SamplingParams
+  ): Branch;
+
+  /**
+   * Fork this branch to a new sequence
+   *
+   * The child shares the parent's KV prefix in memory (metadata-only under unified KV, no KV buffer copy).
+   * Logits, sampler state, and perplexity tracker are cloned so the child
+   * can diverge independently. Fork from any branch — root or intermediate —
+   * to build arbitrarily deep trees.
+   *
+   * @param newSeqId Sequence ID for the forked branch
+   */
+  fork(newSeqId: number): Branch;
+
+  /** Freeze the current logit distribution into this branch. Essential before fork(). */
+  captureLogits(): void;
+
+  /** Decode a single token, write to KV, and capture resulting logits */
+  decodeAndCaptureOne(token: number): void;
+
+  /** Sample next token from branch's frozen logits snapshot */
+  sample(): number;
+
+  /** Accept token for repeat-penalty tracking */
+  accept(token: number): void;
+
+  /** Discard branch — remove its divergent KV entries and free the handle (use for losers) */
+  prune(): void;
+
+  /** Release handle but keep KV entries intact (use for winners, continue with raw ops) */
+  destroy(): void;
+
+  /**
+   * Reseed the sampler's PRNG for diversity after fork()
+   *
+   * CRITICAL for parallel generation: Without reseeding, all forked branches
+   * produce identical outputs because they share the same PRNG state.
+   *
+   * Only affects stochastic samplers (temperature > 0). Greedy samplers are unchanged.
+   *
+   * @param seed - New seed for the PRNG
+   */
+  reseedSampler(seed: number): void;
+
+  /** Sample next token without advancing state. Inspect before committing. */
+  produce(): Produced;
+
+  /** Accept and advance — write token to KV and update branch state. */
+  commit(token: number): void;
+
+  /** Branch's sequence ID */
+  readonly seqId: number;
+
+  /** Branch's current position */
+  readonly position: number;
+
+  /** Branch's perplexity */
+  readonly perplexity: number;
+
+  /** Internal handle (for debugging) */
+  readonly handle: number;
+
+  /** Whether this branch has been disposed */
+  readonly disposed: boolean;
+}

package/lib/index.js

@@ -23,11 +23,11 @@
  * // Safe logits access (Runtime Borrow Checker pattern)
  * const entropy = withLogits(ctx, (logits) => {
  *   // logits is valid here - use synchronously only!
- *   return computeEntropy(logits);
+ *   return myComputeEntropy(logits);
  * });
  *
  * // Or with native reference implementations (for testing)
- * const nativeEntropy = ctx.computeEntropy();
+ * const entropy = ctx.modelEntropy();
  * const token = ctx.greedySample();
  *
  * // Cleanup
@@ -203,7 +203,14 @@ function withLogits(ctx, fn) {
   return result;
 }
 
+const { Branch } = require('./Branch');
+
 module.exports = {
+  /**
+   * Branch class for parallel generation
+   * @see Branch.create()
+   */
+  Branch,
   /**
    * Create a new inference context
    *

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lloyal-labs/lloyal.node",
-  "version": "1.0.6-alpha",
+  "version": "1.0.7",
   "description": "Node.js client for liblloyal+llama.cpp",
   "main": "lib/index.js",
   "types": "lib/index.d.ts",
@@ -19,6 +19,7 @@
     "test": "npm run test:api && npm run test:e2e",
     "test:api": "node test/api.js",
     "test:e2e": "node test/e2e.js",
+    "test:examples": "node test/examples.js",
     "example": "node examples/chat/chat.mjs"
   },
   "repository": {
@@ -48,22 +49,23 @@
   "devDependencies": {
     "cmake-js": "^7.4.0",
     "glob": "^11.0.0",
-    "typedoc": "^0.27.5"
+    "typedoc": "^0.28.16",
+    "typedoc-rhineai-theme": "^1.2.0"
   },
   "optionalDependencies": {
-    "@lloyal-labs/lloyal.node-darwin-arm64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-darwin-x64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-arm64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-arm64-cuda": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-arm64-vulkan": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-x64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-x64-cuda": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-linux-x64-vulkan": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-arm64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-arm64-vulkan": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-x64": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-x64-cuda": "1.0.6-alpha",
-    "@lloyal-labs/lloyal.node-win32-x64-vulkan": "1.0.6-alpha"
+    "@lloyal-labs/lloyal.node-darwin-arm64": "1.0.7",
+    "@lloyal-labs/lloyal.node-darwin-x64": "1.0.7",
+    "@lloyal-labs/lloyal.node-linux-arm64": "1.0.7",
+    "@lloyal-labs/lloyal.node-linux-arm64-cuda": "1.0.7",
+    "@lloyal-labs/lloyal.node-linux-arm64-vulkan": "1.0.7",
+    "@lloyal-labs/lloyal.node-linux-x64": "1.0.7",
+    "@lloyal-labs/lloyal.node-linux-x64-cuda": "1.0.7",
+    "@lloyal-labs/lloyal.node-linux-x64-vulkan": "1.0.7",
+    "@lloyal-labs/lloyal.node-win32-arm64": "1.0.7",
+    "@lloyal-labs/lloyal.node-win32-arm64-vulkan": "1.0.7",
+    "@lloyal-labs/lloyal.node-win32-x64": "1.0.7",
+    "@lloyal-labs/lloyal.node-win32-x64-cuda": "1.0.7",
+    "@lloyal-labs/lloyal.node-win32-x64-vulkan": "1.0.7"
   },
   "engines": {
     "node": ">=22.0.0"

package/scripts/download-test-models.sh

@@ -26,5 +26,15 @@ else
   echo "  ✓ nomic-embed-text already exists"
 fi
 
+# slim-summary-tool (1.7GB) - Summary sidecar for dynamic sinks
+if [ ! -f "slim-summarize.gguf" ]; then
+  echo "  → Downloading slim-summarize.gguf..."
+  curl -L -o "slim-summarize.gguf" \
+    "https://huggingface.co/llmware/slim-summary-tool/resolve/main/slim-summary-tool.gguf"
+  echo "  ✓ Downloaded slim-summarize"
+else
+  echo "  ✓ slim-summarize already exists"
+fi
+
 echo ""
 echo "✅ All test models ready"