@synsci/cli-darwin-x64 1.1.49

package/bin/skills/speculative-decoding/references/lookahead.md

@@ -0,0 +1,309 @@
+# Lookahead Decoding: Jacobi Iteration
+
+Based on ICML 2024 paper and LMSYS blog post
+
+## Overview
+
+**Source**: https://lmsys.org/blog/2023-11-21-lookahead-decoding/
+**Paper**: ICML 2024
+**GitHub**: https://github.com/hao-ai-lab/LookaheadDecoding
+
+Lookahead Decoding breaks sequential dependency in autoregressive decoding using Jacobi iteration, achieving 1.5-2.3× speedup without draft models or training.
+
+## Core Concept
+
+### Reformulation as Equation Solving
+
+**Traditional autoregressive**:
+```
+y_t = f(x, y_1, y_2, ..., y_{t-1})  # Sequential
+```
+
+**Jacobi iteration**:
+```
+y_t^{(k+1)} = f(x, y_1^{(k)}, y_2^{(k)}, ..., y_{t-1}^{(k)})  # Parallel
+```
+
+**Key insight**: Although exact parallel decoding is impossible, we can generate multiple disjoint n-grams in parallel that may fit into the final sequence.
+
+## Two-Branch Architecture
+
+### Lookahead Branch
+
+**Purpose**: Generate potential token sequences (n-grams) in parallel.
+
+**Parameters**:
+- `W` (window size): How many steps ahead to look
+- `N` (n-gram size): How many past tokens to use for generation
+
+```python
+# Example: W=5, N=3
+# Generate n-grams at positions 1-5 using past 1-3 tokens
+
+def lookahead_branch(model, tokens, W=5, N=3):
+    """Generate n-grams using Jacobi iteration."""
+    candidates = {}
+
+    for w in range(1, W + 1):         # Position offset
+        for n in range(1, N + 1):     # N-gram length
+            # Use n past tokens to predict at position w
+            past_tokens = tokens[-n:]
+            future_position = len(tokens) + w
+
+            # Generate n-gram
+            ngram = model.generate_ngram(
+                context=past_tokens,
+                position=future_position,
+                length=n
+            )
+
+            candidates[(w, n)] = ngram
+
+    return candidates
+```
+
+**Output**: Pool of candidate n-grams that might match future sequence.
+
+### Verification Branch
+
+**Purpose**: Identify and confirm promising n-grams.
+
+```python
+def verification_branch(model, tokens, candidates):
+    """Verify which candidates match actual sequence."""
+    verified = []
+
+    for ngram in candidates:
+        # Check if ngram's first token matches last generated token
+        if ngram[0] == tokens[-1]:
+            # Verify full n-gram with model
+            is_valid = model.verify_sequence(tokens + ngram)
+
+            if is_valid:
+                verified.append(ngram)
+
+    # Return longest verified n-gram
+    return max(verified, key=len) if verified else None
+```
+
+**Acceptance**: N-gram accepted if its first token matches the last input token and model confirms the sequence.
+
+## Algorithm
+
+### Complete Lookahead Decoding
+
+```python
+class LookaheadDecoding:
+    def __init__(self, model, W=15, N=5, G=5):
+        """
+        Args:
+            W: Window size (lookahead distance)
+            N: N-gram size (context length)
+            G: Guess size (parallel candidates)
+        """
+        self.model = model
+        self.W = W
+        self.N = N
+        self.G = G
+
+    def generate(self, input_ids, max_new_tokens=256):
+        tokens = input_ids.clone()
+
+        while len(tokens) < max_new_tokens:
+            # 1. Lookahead: Generate candidates
+            candidates = self._lookahead_step(tokens)
+
+            # 2. Verification: Find matching n-grams
+            accepted_ngram = self._verification_step(tokens, candidates)
+
+            if accepted_ngram is not None:
+                # Accept multiple tokens
+                tokens = torch.cat([tokens, accepted_ngram])
+            else:
+                # Fallback: Generate single token
+                next_token = self.model.generate_next(tokens)
+                tokens = torch.cat([tokens, next_token])
+
+        return tokens
+
+    def _lookahead_step(self, tokens):
+        """Generate candidate n-grams in parallel."""
+        candidates = []
+
+        for w in range(1, self.W + 1):
+            for n in range(1, self.N + 1):
+                # Sample n-gram from model
+                ngram = self.model.sample_ngram(
+                    tokens=tokens,
+                    offset=w,
+                    context_size=n,
+                    num_samples=self.G
+                )
+                candidates.extend(ngram)
+
+        return candidates
+
+    def _verification_step(self, tokens, candidates):
+        """Verify candidates and select best."""
+        valid_ngrams = []
+
+        for ngram in candidates:
+            # Must match continuation
+            if ngram[0] == self._get_next_token_prediction(tokens):
+                # Verify full sequence
+                if self._verify_ngram(tokens, ngram):
+                    valid_ngrams.append(ngram)
+
+        # Return longest valid n-gram
+        return max(valid_ngrams, key=len) if valid_ngrams else None
+```
+
+## Performance Analysis
+
+### Speedup vs Parameters
+
+**From paper (7B model on HumanEval)**:
+
+| Window (W) | N-gram (N) | Speedup | Throughput |
+|------------|------------|---------|------------|
+| 5 | 3 | 1.5× | 45 tokens/sec |
+| 10 | 5 | 1.8× | 54 tokens/sec |
+| 15 | 5 | 2.2× | 66 tokens/sec |
+| 20 | 7 | 2.3× | 69 tokens/sec |
+
+**Hardware configurations (A100 GPU)**:
+
+| Model Size | Recommended W | Recommended N |
+|------------|---------------|---------------|
+| 7B | 15 | 5 |
+| 13B | 10 | 5 |
+| 33B | 7 | 5 |
+| 70B | 5 | 3 |
+
+**Rule**: Larger models → smaller W, N (more expensive to verify)
+
+### Scaling Law
+
+**Key finding from paper**:
+
+"When n-gram size is sufficiently large, exponentially increasing future token guesses can linearly reduce decoding steps."
+
+```
+Speedup ≈ 1 + (W × acceptance_rate)
+
+where acceptance_rate depends on:
+- Model quality (better models = higher acceptance)
+- Task type (code generation > chat)
+- N-gram size (larger N = higher acceptance but more compute)
+```
+
+## Hyperparameter Tuning
+
+### Window Size (W)
+
+```python
+# Trade-off: Larger W = more candidates but more verification cost
+
+W = 5   # Conservative (low overhead, moderate speedup)
+W = 10  # Balanced
+W = 15  # Standard (from paper, 7B models)
+W = 20  # Aggressive (diminishing returns)
+
+# Rule: W should be ~2-3× average token acceptance length
+```
+
+### N-gram Size (N)
+
+```python
+# Trade-off: Larger N = better context but slower generation
+
+N = 3   # Fast generation, less context
+N = 5   # Standard (from paper)
+N = 7   # Better context, slower
+
+# Rule: N should be large enough to capture local patterns
+```
+
+### Guess Size (G)
+
+```python
+# Number of parallel n-gram candidates per position
+
+G = 1   # Deterministic (fastest, lower acceptance)
+G = 5   # Standard (good balance)
+G = 10  # More exploration (higher acceptance, more compute)
+```
+
+## Comparison with Other Methods
+
+| Method | Speedup | Training | Draft Model | Memory |
+|--------|---------|----------|-------------|---------|
+| **Lookahead** | 1.5-2.3× | None | No | Base only |
+| Draft Speculative | 1.5-2× | None | Yes | Base + draft |
+| Medusa | 2-3.6× | Minimal | No | Base + heads |
+
+**Advantages of Lookahead**:
+- Zero training required
+- No draft model needed
+- Works out-of-the-box with any model
+- No model modification
+
+**Disadvantages**:
+- Lower speedup than Medusa
+- More complex implementation
+- Verification overhead
+
+## Task-Specific Performance
+
+**From paper**:
+
+| Task | Baseline | Lookahead | Speedup |
+|------|----------|-----------|---------|
+| **HumanEval (code)** | 30 tok/s | 69 tok/s | 2.3× |
+| **MT-Bench (chat)** | 35 tok/s | 56 tok/s | 1.6× |
+| **GSM8K (math)** | 32 tok/s | 54 tok/s | 1.7× |
+
+**Why code is faster**: Higher n-gram predictability (syntax, patterns).
+
+## Production Deployment
+
+### Integration Example
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Load model
+model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
+tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
+
+# Initialize Lookahead
+lookahead = LookaheadDecoding(
+    model=model,
+    W=15,  # Window size
+    N=5,   # N-gram size
+    G=5    # Guess size
+)
+
+# Generate
+prompt = "Write a Python function to calculate fibonacci:"
+input_ids = tokenizer.encode(prompt, return_tensors="pt")
+
+output = lookahead.generate(input_ids, max_new_tokens=256)
+response = tokenizer.decode(output[0], skip_special_tokens=True)
+
+print(response)
+```
+
+### Optimization Tips
+
+1. **Batch processing**: Verify multiple n-grams in single forward pass
+2. **Caching**: Reuse KV cache across verification steps
+3. **Early stopping**: Stop generation when no candidates match
+4. **Adaptive parameters**: Adjust W, N based on acceptance rate
+
+## Resources
+
+- **Blog Post**: https://lmsys.org/blog/2023-11-21-lookahead-decoding/
+- **GitHub**: https://github.com/hao-ai-lab/LookaheadDecoding
+- **Paper**: ICML 2024 (Break the Sequential Dependency of LLM Inference Using Lookahead Decoding)
+- **NVIDIA Blog**: https://developer.nvidia.com/blog/optimizing-qwen2-5-coder-throughput-with-nvidia-tensorrt-llm-lookahead-decoding/

package/bin/skills/speculative-decoding/references/medusa.md

@@ -0,0 +1,350 @@
+# Medusa: Multiple Decoding Heads
+
+Based on arXiv 2401.10774 (2024) - MEDUSA: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
+
+## Overview
+
+**Source**: https://arxiv.org/abs/2401.10774
+**GitHub**: https://github.com/FasterDecoding/Medusa
+
+Medusa augments LLM inference by adding extra decoding heads to predict multiple subsequent tokens in parallel, achieving 2.2-3.6× speedup without quality loss.
+
+## Architecture
+
+### Core Innovation
+
+Instead of separate draft model, add multiple prediction heads to existing LLM:
+
+```
+Input → Base LLM (frozen or fine-tuned) → Hidden State
+                                             ├→ Head 0 (original, predicts t+1)
+                                             ├→ Head 1 (predicts t+2)
+                                             ├→ Head 2 (predicts t+3)
+                                             └→ Head 3 (predicts t+4)
+```
+
+### Tree-Based Attention
+
+**Key mechanism**: Construct candidate tree, verify all paths in single forward pass.
+
+Example with 2 heads, top-2 candidates per head:
+
+```
+                Root (current token)
+                /                  \
+           Candidate 1a         Candidate 1b    (Head 1: 2 options)
+           /        \           /        \
+        C2a        C2b       C2c        C2d     (Head 2: 4 total paths)
+```
+
+Single forward pass evaluates entire tree (4 candidates) in parallel!
+
+## Training Methods
+
+### Medusa-1: Frozen Backbone
+
+**Approach**: Keep base LLM frozen, train only Medusa heads.
+
+**Advantages**:
+- Lossless (base model unchanged)
+- Fast training (~few hours on 8 GPUs)
+- Minimal data needed (~10M tokens)
+
+**Performance**: 2.2× speedup
+
+```python
+# Training loop for Medusa-1
+for batch in dataloader:
+    # Frozen base model
+    with torch.no_grad():
+        hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]
+
+    # Train Medusa heads
+    for i, head in enumerate(medusa_heads):
+        logits = head(hidden_states)
+        # Target: tokens shifted by (i+1) positions
+        targets = batch['input_ids'][:, i+1:]
+        loss += F.cross_entropy(logits[:, :-i-1], targets)
+
+    loss.backward()
+    optimizer.step()
+```
+
+**Training Data**: Any text corpus (Wikipedia, C4, etc.)
+
+### Medusa-2: Joint Fine-Tuning
+
+**Approach**: Fine-tune base LLM + Medusa heads together.
+
+**Advantages**:
+- Better prediction accuracy (heads aligned with base)
+- Higher speedup (2.3-3.6×)
+
+**Challenge**: Must preserve base model capabilities
+
+**Solution**: Special training recipe:
+1. Start with pre-trained base model
+2. Add Medusa heads
+3. Fine-tune both together with careful LR scheduling
+4. Use high-quality data to avoid degradation
+
+```python
+# Medusa-2 training
+# All parameters trainable
+for param in base_model.parameters():
+    param.requires_grad = True  # Unfreeze base
+
+for param in medusa_heads.parameters():
+    param.requires_grad = True
+
+# Different learning rates
+optimizer = torch.optim.AdamW([
+    {'params': base_model.parameters(), 'lr': 1e-5},      # Lower for base
+    {'params': medusa_heads.parameters(), 'lr': 1e-3},    # Higher for heads
+])
+```
+
+**Performance**: 2.3-3.6× speedup
+
+## Inference Algorithm
+
+### Candidate Generation
+
+```python
+def medusa_generate_candidates(base_logits, medusa_head_logits, top_k=10):
+    """Generate candidate sequences using tree structure."""
+    candidates = []
+
+    # Base token (original LLM output)
+    base_token = torch.argmax(base_logits, dim=-1)
+
+    # For each Medusa head, get top-k predictions
+    medusa_candidates = []
+    for head_logits in medusa_head_logits:
+        top_k_tokens = torch.topk(head_logits, k=top_k, dim=-1).indices
+        medusa_candidates.append(top_k_tokens)
+
+    # Build candidate tree (all combinations)
+    # With 4 heads, top-2 each: 2^4 = 16 candidates
+    for combo in itertools.product(*medusa_candidates):
+        candidate = [base_token] + list(combo)
+        candidates.append(candidate)
+
+    return candidates  # Shape: (num_candidates, seq_len)
+```
+
+### Tree Verification
+
+```python
+def medusa_verify_candidates(model, candidates, past_key_values):
+    """Verify all candidates in single forward pass using tree attention."""
+    # Construct tree attention mask
+    # All candidates share prefix, diverge at different points
+    attention_mask = build_tree_attention_mask(candidates)
+
+    # Single forward pass for all candidates
+    outputs = model(
+        input_ids=candidates,
+        attention_mask=attention_mask,
+        past_key_values=past_key_values,
+        use_cache=True
+    )
+
+    # Score each candidate
+    scores = compute_acceptance_scores(outputs.logits, candidates)
+
+    # Accept longest valid candidate
+    best_candidate = select_best(candidates, scores)
+
+    return best_candidate
+```
+
+### Acceptance Criterion
+
+**Posterior threshold**: Accept token if probability exceeds threshold.
+
+```python
+def should_accept(token, token_prob, threshold=0.09):
+    """Medusa acceptance criterion."""
+    return token_prob >= threshold
+
+# Typical thresholds:
+# - 0.09: Standard (from paper)
+# - 0.05: Conservative (fewer rejections, slower)
+# - 0.15: Aggressive (more rejections, faster when works)
+```
+
+## Performance Results
+
+**From paper (Vicuna-7B, MT-Bench):**
+
+| Configuration | Speedup | Quality (MT-Bench score) |
+|---------------|---------|--------------------------|
+| Baseline | 1.0× | 6.57 |
+| Medusa-1 (frozen) | 2.2× | 6.57 (lossless) |
+| Medusa-2 (joint) | 2.3× | 6.60 (+0.03) |
+| Medusa-2 (optimized) | 3.6× | 6.55 (-0.02) |
+
+**Key findings**:
+- Medusa-1: No quality degradation (frozen base)
+- Medusa-2: Slight quality improvement possible
+- Trade-off: More aggressive = faster but may reduce quality
+
+## Hyperparameter Tuning
+
+### Number of Heads
+
+```python
+# Typical configurations:
+num_heads = 2  # Conservative (2× speedup)
+num_heads = 3  # Balanced (2.5× speedup)
+num_heads = 4  # Standard (3× speedup, from paper)
+num_heads = 5  # Aggressive (3.5×+ speedup)
+
+# Rule: More heads = more candidates but also more computation
+# Optimal: 3-4 heads for most models
+```
+
+### Top-K per Head
+
+```python
+# Candidates per head
+top_k = 2   # Standard (2^num_heads total candidates)
+top_k = 3   # More candidates (3^num_heads)
+top_k = 5   # Many candidates (5^num_heads)
+
+# Example with 4 heads:
+# top_k=2: 16 candidates (fast)
+# top_k=3: 81 candidates (slower verification)
+```
+
+### Tree Construction
+
+**Medusa Choices** (which candidate paths to explore):
+
+```python
+# Standard configuration (from paper)
+medusa_choices = [
+    [0],        # Only head 0
+    [0, 0],     # Head 0, then head 1 (first candidate)
+    [0, 1],     # Head 0, then head 1 (second candidate)
+    [0, 0, 0],  # All heads (first path)
+]
+
+# Aggressive configuration (more paths)
+medusa_choices = [
+    [0],
+    [0, 0], [0, 1],
+    [0, 0, 0], [0, 0, 1], [0, 1, 0], [0, 1, 1],
+]
+```
+
+## Training Recipe
+
+### Data Requirements
+
+**Medusa-1**:
+- Amount: 10M-100M tokens
+- Quality: Any text corpus works
+- Time: 2-8 hours on 8× A100
+
+**Medusa-2**:
+- Amount: 100M-1B tokens
+- Quality: High-quality (same domain as target use case)
+- Time: 1-3 days on 8× A100
+
+### Training Script
+
+```bash
+# Clone Medusa repo
+git clone https://github.com/FasterDecoding/Medusa
+cd Medusa
+
+# Train Medusa-1 (frozen base)
+python medusa/train/train.py \
+    --model_name_or_path lmsys/vicuna-7b-v1.3 \
+    --data_path ShareGPT_Vicuna_unfiltered/ShareGPT_V4.3_unfiltered_cleaned_split.json \
+    --bf16 True \
+    --output_dir medusa-vicuna-7b-v1.3 \
+    --num_train_epochs 3 \
+    --per_device_train_batch_size 4 \
+    --gradient_accumulation_steps 8 \
+    --learning_rate 1e-3 \
+    --medusa_num_heads 4 \
+    --medusa_num_layers 1 \
+    --freeze_base_model True  # Medusa-1
+
+# Train Medusa-2 (joint fine-tuning)
+python medusa/train/train.py \
+    --model_name_or_path lmsys/vicuna-7b-v1.3 \
+    --data_path high_quality_data.json \
+    --bf16 True \
+    --output_dir medusa-vicuna-7b-v1.3-joint \
+    --num_train_epochs 1 \
+    --per_device_train_batch_size 4 \
+    --gradient_accumulation_steps 8 \
+    --learning_rate 1e-5 \  # Lower LR for base model
+    --medusa_num_heads 4 \
+    --freeze_base_model False  # Medusa-2 (joint)
+```
+
+## Deployment
+
+### Loading Medusa Model
+
+```python
+from medusa.model.medusa_model import MedusaModel
+
+# Load pre-trained Medusa model
+model = MedusaModel.from_pretrained(
+    "FasterDecoding/medusa-vicuna-7b-v1.3",
+    torch_dtype=torch.float16,
+    device_map="auto"
+)
+
+# Or load base + Medusa heads separately
+base_model = AutoModelForCausalLM.from_pretrained("lmsys/vicuna-7b-v1.3")
+medusa_heads = torch.load("medusa_heads.pt")
+model = MedusaModel(base_model, medusa_heads)
+```
+
+### Generation
+
+```python
+# Generate with Medusa
+outputs = model.medusa_generate(
+    input_ids,
+    max_new_tokens=256,
+    temperature=0.7,
+    posterior_threshold=0.09,    # Acceptance threshold
+    posterior_alpha=0.3,         # Tree construction parameter
+    medusa_choices=medusa_choices,  # Candidate paths
+)
+```
+
+## Comparison with Speculative Decoding
+
+| Aspect | Medusa | Speculative Decoding |
+|--------|--------|----------------------|
+| **Draft Model** | Built-in (heads) | External (separate model) |
+| **Training** | Minimal (heads only) | None (use existing small model) |
+| **Memory** | Base + heads (~1-2% overhead) | Base + draft (can be large) |
+| **Speedup** | 2-3.6× | 1.5-2× |
+| **Deployment** | Single model | Two models |
+
+**When to use Medusa**:
+- Want single model deployment
+- Can afford minimal training
+- Need best speedup (3×+)
+
+**When to use Speculative**:
+- Have existing small model
+- Zero training budget
+- Simpler setup
+
+## Resources
+
+- **Paper**: https://arxiv.org/abs/2401.10774
+- **GitHub**: https://github.com/FasterDecoding/Medusa
+- **Blog**: https://www.together.ai/blog/medusa
+- **Demo**: https://sites.google.com/view/medusa-llm