@r1z4x/tezgah 1.0.0

package/skills/ai-research/01-model-architecture/mamba/references/benchmarks.md

@@ -0,0 +1,256 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 01-model-architecture/mamba/references/benchmarks.md; MIT (c) Orchestra Research -->
+# Mamba Performance Benchmarks
+
+## Inference Speed Comparison
+
+### Throughput (tokens/sec)
+
+**Mamba-1.4B vs Transformer-1.3B** on single A100 80GB:
+
+| Sequence Length | Mamba-1.4B | Transformer-1.3B | Speedup |
+|----------------|------------|------------------|---------|
+| 512 | 8,300 | 6,200 | 1.3× |
+| 1024 | 7,800 | 4,100 | 1.9× |
+| 2048 | 7,200 | 2,300 | 3.1× |
+| 4096 | 6,800 | 1,200 | 5.7× |
+| 8192 | 6,400 | 600 | **10.7×** |
+| 16384 | 6,100 | OOM | ∞ |
+
+**Key insight**: Speedup grows with sequence length (Mamba O(n) vs Transformer O(n²))
+
+### Latency (ms per token)
+
+**Generation latency** (batch size 1, autoregressive):
+
+| Model | First Token | Per Token | 100 Tokens Total |
+|-------|-------------|-----------|------------------|
+| Mamba-130M | 3 ms | 0.8 ms | 83 ms |
+| Transformer-130M | 5 ms | 1.2 ms | 125 ms |
+| Mamba-1.4B | 12 ms | 3.2 ms | 332 ms |
+| Transformer-1.3B | 18 ms | 8.5 ms | 868 ms |
+| Mamba-2.8B | 20 ms | 6.1 ms | 631 ms |
+| Transformer-2.7B | 35 ms | 18.2 ms | 1855 ms |
+
+**Mamba advantage**: Constant per-token latency regardless of context length
+
+## Memory Usage
+
+### Training Memory (BF16, per GPU)
+
+**Mamba-1.4B** training memory breakdown:
+
+| Sequence Length | Activations | Gradients | Optimizer | Total | vs Transformer |
+|----------------|-------------|-----------|-----------|-------|----------------|
+| 512 | 2.1 GB | 3.2 GB | 11.2 GB | 16.5 GB | 0.9× |
+| 1024 | 3.8 GB | 3.2 GB | 11.2 GB | 18.2 GB | 0.6× |
+| 2048 | 7.2 GB | 3.2 GB | 11.2 GB | 21.6 GB | 0.4× |
+| 4096 | 14.1 GB | 3.2 GB | 11.2 GB | 28.5 GB | 0.25× |
+| 8192 | 28.0 GB | 3.2 GB | 11.2 GB | 42.4 GB | 0.15× |
+
+**Note**: Transformer OOMs at 8K sequence length on 40GB A100
+
+### Inference Memory (FP16, batch size 1)
+
+| Model | KV Cache (8K ctx) | State (Mamba) | Ratio |
+|-------|------------------|---------------|-------|
+| 130M | 2.1 GB | 0 MB | ∞ |
+| 370M | 5.2 GB | 0 MB | ∞ |
+| 1.4B | 19.7 GB | 0 MB | ∞ |
+| 2.8B | 38.4 GB | 0 MB | ∞ |
+
+**Mamba stores no KV cache** - constant memory per token!
+
+Actual Mamba state size:
+- 130M: ~3 MB (d_model × d_state × n_layers = 768 × 16 × 24)
+- 2.8B: ~13 MB (2560 × 16 × 64)
+
+## Language Modeling Benchmarks
+
+### Perplexity on Common Datasets
+
+**Models trained on The Pile (300B tokens)**:
+
+| Model | Params | Pile (val) | WikiText-103 | C4 | Lambada |
+|-------|--------|------------|--------------|-----|---------|
+| Pythia | 160M | 29.6 | 28.4 | 23.1 | 51.2 |
+| **Mamba** | **130M** | **28.1** | **26.7** | **21.8** | **48.3** |
+| Pythia | 410M | 18.3 | 17.6 | 16.2 | 32.1 |
+| **Mamba** | **370M** | **16.7** | **16.2** | **15.1** | **28.4** |
+| Pythia | 1.4B | 10.8 | 10.2 | 11.3 | 15.2 |
+| **Mamba** | **1.4B** | **9.1** | **9.6** | **10.1** | **12.8** |
+| Pythia | 2.8B | 8.3 | 7.9 | 9.2 | 10.6 |
+| **Mamba** | **2.8B** | **7.4** | **7.2** | **8.3** | **9.1** |
+
+**Mamba consistently outperforms** Transformers of similar size by 10-20%
+
+### Zero-Shot Task Performance
+
+**Mamba-2.8B vs Transformer-2.7B** on common benchmarks:
+
+| Task | Mamba-2.8B | Transformer-2.7B | Delta |
+|------|------------|------------------|-------|
+| HellaSwag | 61.3 | 58.7 | +2.6 |
+| PIQA | 78.1 | 76.4 | +1.7 |
+| ARC-Easy | 68.2 | 65.9 | +2.3 |
+| ARC-Challenge | 42.7 | 40.1 | +2.6 |
+| WinoGrande | 64.8 | 62.3 | +2.5 |
+| OpenBookQA | 43.2 | 41.8 | +1.4 |
+| BoolQ | 71.4 | 68.2 | +3.2 |
+| MMLU (5-shot) | 35.2 | 33.8 | +1.4 |
+
+**Average improvement**: +2.2 points across benchmarks
+
+## Audio Modeling Benchmarks
+
+### SC09 (Speech Commands)
+
+**Task**: Audio classification (10 classes)
+
+| Model | Params | Accuracy | Inference (ms) |
+|-------|--------|----------|----------------|
+| Transformer | 8.2M | 96.2% | 18 ms |
+| S4 | 6.1M | 97.1% | 8 ms |
+| **Mamba** | **6.3M** | **98.4%** | **6 ms** |
+
+### LJSpeech (Speech Generation)
+
+**Task**: Text-to-speech quality (MOS score)
+
+| Model | Params | MOS ↑ | RTF ↓ |
+|-------|--------|-------|-------|
+| Transformer | 12M | 3.82 | 0.45 |
+| Conformer | 11M | 3.91 | 0.38 |
+| **Mamba** | **10M** | **4.03** | **0.21** |
+
+**RTF** (Real-Time Factor): Lower is better (0.21 = 5× faster than real-time)
+
+## Genomics Benchmarks
+
+### Human Reference Genome (HG38)
+
+**Task**: Next nucleotide prediction
+
+| Model | Context Length | Perplexity | Throughput |
+|-------|----------------|------------|------------|
+| Transformer | 1024 | 3.21 | 1,200 bp/s |
+| Hyena | 32768 | 2.87 | 8,500 bp/s |
+| **Mamba** | **1M** | **2.14** | **45,000 bp/s** |
+
+**Mamba handles million-length sequences** efficiently
+
+## Scaling Laws
+
+### Compute-Optimal Training
+
+**FLOPs vs perplexity** (The Pile validation):
+
+| Model Size | Training FLOPs | Mamba Perplexity | Transformer Perplexity |
+|------------|----------------|------------------|------------------------|
+| 130M | 6e19 | 28.1 | 29.6 |
+| 370M | 3e20 | 16.7 | 18.3 |
+| 790M | 8e20 | 12.3 | 13.9 |
+| 1.4B | 2e21 | 9.1 | 10.8 |
+| 2.8B | 6e21 | 7.4 | 8.3 |
+
+**Scaling coefficient**: Mamba achieves same perplexity as Transformer with **0.8×** compute
+
+### Parameter Efficiency
+
+**Perplexity 10.0 target** on The Pile:
+
+| Model Type | Parameters Needed | Memory (inference) |
+|------------|-------------------|-------------------|
+| Transformer | 1.6B | 3.2 GB |
+| **Mamba** | **1.1B** | **2.2 GB** |
+
+**Mamba needs ~30% fewer parameters** for same performance
+
+## Long-Range Arena (LRA)
+
+**Task**: Long-context understanding benchmarks
+
+| Task | Length | Transformer | S4 | Mamba |
+|------|--------|-------------|-----|-------|
+| ListOps | 2K | 36.4% | 59.6% | **61.2%** |
+| Text | 4K | 64.3% | 86.8% | **88.1%** |
+| Retrieval | 4K | 57.5% | 90.9% | **92.3%** |
+| Image | 1K | 42.4% | 88.7% | **89.4%** |
+| PathFinder | 1K | 71.4% | 86.1% | **87.8%** |
+| Path-X | 16K | OOM | 88.3% | **91.2%** |
+
+**Average**: Mamba 85.0%, S4 83.4%, Transformer 54.4%
+
+## Training Throughput
+
+### Tokens/sec During Training
+
+**8× A100 80GB** cluster, BF16, different sequence lengths:
+
+| Model | Seq Len 512 | Seq Len 2K | Seq Len 8K | Seq Len 32K |
+|-------|-------------|------------|------------|-------------|
+| Transformer-1.3B | 180K | 52K | OOM | OOM |
+| **Mamba-1.4B** | **195K** | **158K** | **121K** | **89K** |
+| Transformer-2.7B | 92K | 26K | OOM | OOM |
+| **Mamba-2.8B** | **98K** | **81K** | **62K** | **45K** |
+
+**Mamba scales to longer sequences** without OOM
+
+## Hardware Utilization
+
+### GPU Memory Bandwidth
+
+**Mamba-1.4B** inference on different GPUs:
+
+| GPU | Memory BW | Tokens/sec | Efficiency |
+|-----|-----------|------------|------------|
+| A100 80GB | 2.0 TB/s | 6,800 | 85% |
+| A100 40GB | 1.6 TB/s | 5,400 | 84% |
+| V100 32GB | 900 GB/s | 3,100 | 86% |
+| RTX 4090 | 1.0 TB/s | 3,600 | 90% |
+
+**High efficiency**: Mamba is memory-bandwidth bound (good!)
+
+### Multi-GPU Scaling
+
+**Mamba-2.8B** training throughput:
+
+| GPUs | Tokens/sec | Scaling Efficiency |
+|------|------------|-------------------|
+| 1× A100 | 12,300 | 100% |
+| 2× A100 | 23,800 | 97% |
+| 4× A100 | 46,100 | 94% |
+| 8× A100 | 89,400 | 91% |
+| 16× A100 | 172,000 | 88% |
+
+**Near-linear scaling** up to 16 GPUs
+
+## Cost Analysis
+
+### Training Cost (USD)
+
+**Training to The Pile perplexity 10.0** on cloud GPUs:
+
+| Model | Cloud GPUs | Hours | Cost (A100) | Cost (H100) |
+|-------|------------|-------|-------------|-------------|
+| Transformer-1.6B | 8× A100 | 280 | $8,400 | $4,200 |
+| **Mamba-1.1B** | **8× A100** | **180** | **$5,400** | **$2,700** |
+
+**Savings**: 36% cost reduction vs Transformer
+
+### Inference Cost (USD/million tokens)
+
+**API-style inference** (batch size 1, 2K context):
+
+| Model | Latency | Cost/M tokens | Quality (perplexity) |
+|-------|---------|---------------|---------------------|
+| Transformer-1.3B | 8.5 ms/tok | $0.42 | 10.8 |
+| **Mamba-1.4B** | **3.2 ms/tok** | **$0.18** | **9.1** |
+
+**Mamba provides**: 2.6× faster, 57% cheaper, better quality
+
+## Resources
+
+- Benchmarks code: https://github.com/state-spaces/mamba/tree/main/benchmarks
+- Paper (Mamba-1): https://arxiv.org/abs/2312.00752 (Section 4: Experiments)
+- Paper (Mamba-2): https://arxiv.org/abs/2405.21060 (Section 5: Experiments)
+- Pretrained models: https://huggingface.co/state-spaces

package/skills/ai-research/01-model-architecture/mamba/references/training-guide.md

@@ -0,0 +1,389 @@
+<!-- vendored from orchestra-research/AI-research-SKILLs@773a52944ba4747a18bd4ae9ade53fff041adcbc 01-model-architecture/mamba/references/training-guide.md; MIT (c) Orchestra Research -->
+# Mamba Training Guide
+
+## Training from Scratch
+
+### Setup Environment
+
+```bash
+# Install dependencies
+pip install torch>=1.12.0 --extra-index-url https://download.pytorch.org/whl/cu116
+pip install packaging ninja
+pip install causal-conv1d>=1.1.0
+pip install mamba-ssm
+
+# Verify CUDA
+python -c "import torch; print(torch.cuda.is_available())"
+```
+
+### Basic Training Loop
+
+```python
+import torch
+from mamba_ssm import Mamba
+from torch.utils.data import DataLoader
+
+# Model setup
+model = Mamba(
+    d_model=512,
+    d_state=16,
+    d_conv=4,
+    expand=2
+).cuda()
+
+# Optimizer (same as GPT)
+optimizer = torch.optim.AdamW(
+    model.parameters(),
+    lr=6e-4,
+    betas=(0.9, 0.95),
+    weight_decay=0.1
+)
+
+# Training loop
+for batch in dataloader:
+    inputs, targets = batch
+    inputs, targets = inputs.cuda(), targets.cuda()
+
+    # Forward
+    logits = model(inputs)
+    loss = F.cross_entropy(logits.view(-1, vocab_size), targets.view(-1))
+
+    # Backward
+    optimizer.zero_grad()
+    loss.backward()
+    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+    optimizer.step()
+```
+
+## Distributed Training
+
+### Single-Node Multi-GPU (DDP)
+
+```python
+import torch.distributed as dist
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+# Initialize process group
+dist.init_process_group("nccl")
+local_rank = int(os.environ["LOCAL_RANK"])
+torch.cuda.set_device(local_rank)
+
+# Wrap model
+model = Mamba(...).cuda()
+model = DDP(model, device_ids=[local_rank])
+
+# Train
+optimizer = torch.optim.AdamW(model.parameters(), lr=6e-4)
+for batch in dataloader:
+    loss = compute_loss(model, batch)
+    optimizer.zero_grad()
+    loss.backward()
+    optimizer.step()
+```
+
+**Launch**:
+```bash
+torchrun --nproc_per_node=8 train.py
+```
+
+### Multi-Node Training
+
+```bash
+# Node 0 (master)
+torchrun --nproc_per_node=8 \
+  --nnodes=4 --node_rank=0 \
+  --master_addr=$MASTER_ADDR --master_port=29500 \
+  train.py
+
+# Node 1-3 (workers)
+torchrun --nproc_per_node=8 \
+  --nnodes=4 --node_rank=$NODE_RANK \
+  --master_addr=$MASTER_ADDR --master_port=29500 \
+  train.py
+```
+
+## Mixed Precision Training
+
+### BF16 (Recommended)
+
+```python
+from torch.cuda.amp import autocast, GradScaler
+
+# BF16 (no scaler needed on A100/H100)
+for batch in dataloader:
+    with autocast(dtype=torch.bfloat16):
+        logits = model(inputs)
+        loss = F.cross_entropy(logits.view(-1, vocab_size), targets.view(-1))
+
+    optimizer.zero_grad()
+    loss.backward()
+    optimizer.step()
+```
+
+### FP16 (with gradient scaling)
+
+```python
+scaler = GradScaler()
+
+for batch in dataloader:
+    with autocast(dtype=torch.float16):
+        logits = model(inputs)
+        loss = F.cross_entropy(logits.view(-1, vocab_size), targets.view(-1))
+
+    optimizer.zero_grad()
+    scaler.scale(loss).backward()
+    scaler.unscale_(optimizer)
+    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+    scaler.step(optimizer)
+    scaler.update()
+```
+
+## Hyperparameter Recommendations
+
+### Learning Rate Schedule
+
+```python
+# Cosine decay with warmup (GPT-3 style)
+def get_lr(it, warmup_iters=2000, lr_decay_iters=600000):
+    max_lr = 6e-4
+    min_lr = 6e-5
+
+    # Warmup
+    if it < warmup_iters:
+        return max_lr * it / warmup_iters
+
+    # Decay
+    if it > lr_decay_iters:
+        return min_lr
+
+    # Cosine
+    decay_ratio = (it - warmup_iters) / (lr_decay_iters - warmup_iters)
+    coeff = 0.5 * (1.0 + math.cos(math.pi * decay_ratio))
+    return min_lr + coeff * (max_lr - min_lr)
+
+# Apply in training loop
+for it, batch in enumerate(dataloader):
+    lr = get_lr(it)
+    for param_group in optimizer.param_groups:
+        param_group['lr'] = lr
+```
+
+### Batch Size Recommendations
+
+| Model Size | Per-GPU Batch | Gradient Accum | Effective Batch | GPUs |
+|------------|---------------|----------------|-----------------|------|
+| 130M | 32 | 4 | 1024 | 8 |
+| 370M | 16 | 8 | 1024 | 8 |
+| 790M | 8 | 8 | 512 | 8 |
+| 1.4B | 4 | 16 | 512 | 8 |
+| 2.8B | 2 | 16 | 256 | 8 |
+
+```python
+# Gradient accumulation
+accumulation_steps = 8
+optimizer.zero_grad()
+
+for i, batch in enumerate(dataloader):
+    loss = compute_loss(model, batch) / accumulation_steps
+    loss.backward()
+
+    if (i + 1) % accumulation_steps == 0:
+        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+        optimizer.step()
+        optimizer.zero_grad()
+```
+
+### Optimizer Configuration
+
+```python
+# AdamW (recommended)
+optimizer = torch.optim.AdamW(
+    model.parameters(),
+    lr=6e-4,           # Peak learning rate
+    betas=(0.9, 0.95), # Standard for LLMs
+    eps=1e-8,
+    weight_decay=0.1   # Important for generalization
+)
+
+# Weight decay exemptions (optional)
+decay = set()
+no_decay = set()
+for name, param in model.named_parameters():
+    if 'norm' in name or 'bias' in name:
+        no_decay.add(param)
+    else:
+        decay.add(param)
+
+optimizer = torch.optim.AdamW([
+    {'params': list(decay), 'weight_decay': 0.1},
+    {'params': list(no_decay), 'weight_decay': 0.0}
+], lr=6e-4, betas=(0.9, 0.95))
+```
+
+## Memory Optimization
+
+### Gradient Checkpointing
+
+```python
+from torch.utils.checkpoint import checkpoint
+
+class MambaBlock(nn.Module):
+    def __init__(self, d_model, use_checkpoint=False):
+        super().__init__()
+        self.use_checkpoint = use_checkpoint
+        self.norm = RMSNorm(d_model)
+        self.mamba = Mamba(d_model)
+
+    def forward(self, x):
+        if self.use_checkpoint and self.training:
+            return x + checkpoint(self._forward, x, use_reentrant=False)
+        return x + self._forward(x)
+
+    def _forward(self, x):
+        return self.mamba(self.norm(x))
+
+# Enable for training
+model = MambaLM(use_checkpoint=True)
+```
+
+**Memory savings**: ~30-40% with minimal speed impact
+
+### Flash Attention Integration
+
+Mamba's CUDA kernels already use flash-attention-style optimizations:
+- Fused operations in single kernel
+- Recomputation in backward pass
+- No intermediate activation storage
+
+## Long Context Training
+
+### Sequence Length Progression
+
+```python
+# Start short, increase gradually
+training_stages = [
+    {'seq_len': 512,  'iters': 50000},
+    {'seq_len': 1024, 'iters': 100000},
+    {'seq_len': 2048, 'iters': 150000},
+    {'seq_len': 4096, 'iters': 200000},
+]
+
+for stage in training_stages:
+    dataloader = create_dataloader(seq_len=stage['seq_len'])
+    train(model, dataloader, max_iters=stage['iters'])
+```
+
+### Memory Requirements (Batch Size 1)
+
+| Sequence Length | 130M Model | 370M Model | 1.4B Model |
+|----------------|------------|------------|------------|
+| 2K | 4 GB | 8 GB | 24 GB |
+| 4K | 5 GB | 10 GB | 32 GB |
+| 8K | 6 GB | 14 GB | 48 GB |
+| 16K | 8 GB | 20 GB | 64 GB |
+| 32K | 12 GB | 32 GB | 96 GB |
+
+**Mamba advantage**: Memory grows **linearly**, Transformers grow **quadratically**
+
+## Common Training Issues
+
+### Issue: OOM during training
+
+**Solution 1**: Reduce batch size
+```python
+per_gpu_batch = 8  # Reduce from 16
+gradient_accumulation = 8  # Increase from 4
+```
+
+**Solution 2**: Enable gradient checkpointing
+```python
+model = MambaLM(use_checkpoint=True)
+```
+
+**Solution 3**: Use smaller sequence length
+```python
+seq_len = 1024  # Reduce from 2048
+```
+
+### Issue: Training unstable (loss spikes)
+
+**Solution 1**: Check gradient norm
+```python
+grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+print(f"Grad norm: {grad_norm}")  # Should be < 10
+```
+
+**Solution 2**: Lower learning rate
+```python
+max_lr = 3e-4  # Reduce from 6e-4
+```
+
+**Solution 3**: Check Δ initialization
+```python
+# Ensure dt_min, dt_max are reasonable
+model = Mamba(
+    d_model=512,
+    dt_min=0.001,  # Not too small
+    dt_max=0.1     # Not too large
+)
+```
+
+### Issue: Slow training speed
+
+**Solution 1**: Verify CUDA kernels installed
+```python
+import mamba_ssm
+print(mamba_ssm.__version__)  # Should have CUDA kernels
+```
+
+**Solution 2**: Use BF16 on A100/H100
+```python
+with autocast(dtype=torch.bfloat16):  # Faster than FP16
+    loss = model(inputs)
+```
+
+**Solution 3**: Increase batch size if possible
+```python
+per_gpu_batch = 16  # Increase from 8 (better GPU utilization)
+```
+
+## Checkpointing
+
+### Save/Load Model
+
+```python
+# Save
+checkpoint = {
+    'model': model.state_dict(),
+    'optimizer': optimizer.state_dict(),
+    'iter': iteration,
+    'config': model_config
+}
+torch.save(checkpoint, f'checkpoint_{iteration}.pt')
+
+# Load
+checkpoint = torch.load('checkpoint_100000.pt')
+model.load_state_dict(checkpoint['model'])
+optimizer.load_state_dict(checkpoint['optimizer'])
+iteration = checkpoint['iter']
+```
+
+### Best Practices
+
+```python
+# Save every N iterations
+if iteration % save_interval == 0:
+    save_checkpoint(model, optimizer, iteration)
+
+# Keep only last K checkpoints
+checkpoints = sorted(glob.glob('checkpoint_*.pt'))
+if len(checkpoints) > keep_last:
+    for ckpt in checkpoints[:-keep_last]:
+        os.remove(ckpt)
+```
+
+## Resources
+
+- Training code: https://github.com/state-spaces/mamba/tree/main/benchmarks
+- Pretrained models: https://huggingface.co/state-spaces
+- CUDA installation: https://github.com/state-spaces/mamba#installation