@synsci/cli-darwin-x64 1.1.49

package/bin/skills/mamba/references/architecture-details.md

@@ -0,0 +1,206 @@
+# Mamba Architecture Details
+
+## Selective State Space Mechanism
+
+Mamba's core innovation is the **Selective SSM (S6)** layer that makes state space model parameters input-dependent.
+
+### How S6 Works
+
+**Traditional SSMs** (non-selective):
+```python
+# Fixed A, B, C matrices for all inputs
+h(t) = A * h(t-1) + B * x(t)  # State update
+y(t) = C * h(t)                # Output
+```
+
+**Mamba's Selective SSM**:
+```python
+# Input-dependent parameters
+B(t) = Linear_B(x(t))  # Selection mechanism
+C(t) = Linear_C(x(t))  # Output projection
+Δ(t) = Linear_Δ(x(t))  # Discretization step
+
+# Selective state update
+h(t) = discretize(A, Δ(t)) * h(t-1) + Δ(t) * B(t) * x(t)
+y(t) = C(t) * h(t)
+```
+
+### Key Advantages
+
+**1. Content-based reasoning**:
+- Can selectively remember or forget based on input
+- Addresses discrete modality weakness of traditional SSMs
+- Example: Remembers important tokens, forgets padding
+
+**2. Input-dependent selection**:
+```python
+# Mamba decides per token what to remember
+if is_important(x(t)):
+    Δ(t) = large_value   # Keep in state
+else:
+    Δ(t) = small_value   # Forget quickly
+```
+
+**3. No attention required**:
+- Replaces O(n²) attention with O(n) state updates
+- State dimension is constant (typically 16)
+
+## Model Configuration
+
+### Core Parameters
+
+```python
+from mamba_ssm import Mamba
+
+model = Mamba(
+    d_model=256,      # Hidden dimension (256, 512, 768, 1024, 2048)
+    d_state=16,       # SSM state dimension (fixed at 16 is optimal)
+    d_conv=4,         # Local convolution width (4 is standard)
+    expand=2,         # Expansion factor (1.5-2.0)
+    dt_rank="auto",   # Rank of Δ projection (auto = d_model / 16)
+    dt_min=0.001,     # Min Δ init (controls forgetting rate)
+    dt_max=0.1,       # Max Δ init
+    dt_init="random", # Δ initialization (random, constant)
+    dt_scale=1.0,     # Δ scaling factor
+    conv_bias=True,   # Use bias in convolution
+    bias=False        # Use bias in linear projections
+)
+```
+
+### Parameter Impact
+
+**d_state** (SSM state dimension):
+- Standard: 16 (optimal from ablations)
+- Smaller (8): Faster but less capacity
+- Larger (32, 64): Minimal improvement, 2× slower
+
+**expand** (block expansion):
+- Standard: 2.0
+- Range: 1.5-2.0
+- Controls inner dimension = expand * d_model
+
+**d_conv** (convolution width):
+- Standard: 4
+- Local context window before SSM
+- Helps with positional information
+
+**dt_rank** (Δ projection rank):
+- Auto: d_model / 16 (recommended)
+- Controls Δ parameter efficiency
+- Lower rank = more efficient but less expressive
+
+## Mamba Block Structure
+
+```python
+# Mamba block (replaces Transformer block)
+class MambaBlock(nn.Module):
+    def __init__(self, d_model):
+        self.norm = RMSNorm(d_model)
+        self.mamba = Mamba(d_model, d_state=16, d_conv=4, expand=2)
+
+    def forward(self, x):
+        return x + self.mamba(self.norm(x))  # Residual
+
+# Full model (stack of Mamba blocks)
+model = nn.Sequential(
+    Embedding(...),
+    *[MambaBlock(d_model) for _ in range(n_layers)],
+    RMSNorm(d_model),
+    LMHead(...)
+)
+```
+
+**Key differences from Transformers**:
+- No multi-head attention (MHA)
+- No feedforward network (FFN)
+- Single Mamba layer per block
+- 2× more layers than equivalent Transformer
+
+## Hardware-Aware Implementation
+
+### Parallel Algorithm
+
+Mamba uses a **scan-based parallel algorithm** for training:
+
+```python
+# Parallel mode (training)
+# GPU kernel fuses operations
+y = parallel_scan(A, B, C, x)  # O(n log n) parallel
+
+# Sequential mode (inference)
+# Constant memory RNN-style
+h = 0
+for x_t in sequence:
+    h = A*h + B*x_t
+    y_t = C*h
+```
+
+### Memory Efficiency
+
+**Training**:
+- Recomputes activations in backward pass
+- Similar to FlashAttention strategy
+- Memory: O(batch_size * seq_len * d_model)
+
+**Inference**:
+- RNN-style sequential processing
+- State size: O(d_model * d_state) = constant
+- No KV cache needed (huge advantage!)
+
+### CUDA Kernel Optimizations
+
+```python
+# Fused kernel operations
+- Discretization (continuous → discrete A, B)
+- SSM recurrence (parallel scan)
+- Convolution (efficient 1D conv)
+- All in single GPU kernel
+```
+
+## Layer Count Scaling
+
+Mamba models use **2× layers** compared to Transformers:
+
+| Model | d_model | n_layers | Params |
+|-------|---------|----------|--------|
+| Mamba-130M | 768 | 24 | 130M |
+| Mamba-370M | 1024 | 48 | 370M |
+| Mamba-790M | 1536 | 48 | 790M |
+| Mamba-1.4B | 2048 | 48 | 1.4B |
+| Mamba-2.8B | 2560 | 64 | 2.8B |
+
+**Why 2× layers?**
+- Mamba blocks are simpler (no MHA, no FFN)
+- ~50% fewer parameters per layer
+- Doubling layers matches compute budget
+
+## Initialization Strategy
+
+```python
+# Δ (discretization step) initialization
+dt_init_floor = 1e-4
+dt = torch.exp(
+    torch.rand(d_inner) * (math.log(dt_max) - math.log(dt_min))
+    + math.log(dt_min)
+).clamp(min=dt_init_floor)
+
+# A (state transition) initialization
+A = -torch.exp(torch.rand(d_inner, d_state))  # Negative for stability
+
+# B, C (input/output) initialization
+B = torch.randn(d_inner, d_state)
+C = torch.randn(d_inner, d_state)
+```
+
+**Critical for stability**:
+- A must be negative (exponential decay)
+- Δ in range [dt_min, dt_max]
+- Random initialization helps diversity
+
+## Resources
+
+- Paper: https://arxiv.org/abs/2312.00752 (Mamba-1)
+- Paper: https://arxiv.org/abs/2405.21060 (Mamba-2)
+- GitHub: https://github.com/state-spaces/mamba
+- Models: https://huggingface.co/state-spaces
+- CUDA kernels: https://github.com/state-spaces/mamba/tree/main/csrc

package/bin/skills/mamba/references/benchmarks.md

@@ -0,0 +1,255 @@
+# Mamba Performance Benchmarks
+
+## Inference Speed Comparison
+
+### Throughput (tokens/sec)
+
+**Mamba-1.4B vs Transformer-1.3B** on single A100 80GB:
+
+| Sequence Length | Mamba-1.4B | Transformer-1.3B | Speedup |
+|----------------|------------|------------------|---------|
+| 512 | 8,300 | 6,200 | 1.3× |
+| 1024 | 7,800 | 4,100 | 1.9× |
+| 2048 | 7,200 | 2,300 | 3.1× |
+| 4096 | 6,800 | 1,200 | 5.7× |
+| 8192 | 6,400 | 600 | **10.7×** |
+| 16384 | 6,100 | OOM | ∞ |
+
+**Key insight**: Speedup grows with sequence length (Mamba O(n) vs Transformer O(n²))
+
+### Latency (ms per token)
+
+**Generation latency** (batch size 1, autoregressive):
+
+| Model | First Token | Per Token | 100 Tokens Total |
+|-------|-------------|-----------|------------------|
+| Mamba-130M | 3 ms | 0.8 ms | 83 ms |
+| Transformer-130M | 5 ms | 1.2 ms | 125 ms |
+| Mamba-1.4B | 12 ms | 3.2 ms | 332 ms |
+| Transformer-1.3B | 18 ms | 8.5 ms | 868 ms |
+| Mamba-2.8B | 20 ms | 6.1 ms | 631 ms |
+| Transformer-2.7B | 35 ms | 18.2 ms | 1855 ms |
+
+**Mamba advantage**: Constant per-token latency regardless of context length
+
+## Memory Usage
+
+### Training Memory (BF16, per GPU)
+
+**Mamba-1.4B** training memory breakdown:
+
+| Sequence Length | Activations | Gradients | Optimizer | Total | vs Transformer |
+|----------------|-------------|-----------|-----------|-------|----------------|
+| 512 | 2.1 GB | 3.2 GB | 11.2 GB | 16.5 GB | 0.9× |
+| 1024 | 3.8 GB | 3.2 GB | 11.2 GB | 18.2 GB | 0.6× |
+| 2048 | 7.2 GB | 3.2 GB | 11.2 GB | 21.6 GB | 0.4× |
+| 4096 | 14.1 GB | 3.2 GB | 11.2 GB | 28.5 GB | 0.25× |
+| 8192 | 28.0 GB | 3.2 GB | 11.2 GB | 42.4 GB | 0.15× |
+
+**Note**: Transformer OOMs at 8K sequence length on 40GB A100
+
+### Inference Memory (FP16, batch size 1)
+
+| Model | KV Cache (8K ctx) | State (Mamba) | Ratio |
+|-------|------------------|---------------|-------|
+| 130M | 2.1 GB | 0 MB | ∞ |
+| 370M | 5.2 GB | 0 MB | ∞ |
+| 1.4B | 19.7 GB | 0 MB | ∞ |
+| 2.8B | 38.4 GB | 0 MB | ∞ |
+
+**Mamba stores no KV cache** - constant memory per token!
+
+Actual Mamba state size:
+- 130M: ~3 MB (d_model × d_state × n_layers = 768 × 16 × 24)
+- 2.8B: ~13 MB (2560 × 16 × 64)
+
+## Language Modeling Benchmarks
+
+### Perplexity on Common Datasets
+
+**Models trained on The Pile (300B tokens)**:
+
+| Model | Params | Pile (val) | WikiText-103 | C4 | Lambada |
+|-------|--------|------------|--------------|-----|---------|
+| Pythia | 160M | 29.6 | 28.4 | 23.1 | 51.2 |
+| **Mamba** | **130M** | **28.1** | **26.7** | **21.8** | **48.3** |
+| Pythia | 410M | 18.3 | 17.6 | 16.2 | 32.1 |
+| **Mamba** | **370M** | **16.7** | **16.2** | **15.1** | **28.4** |
+| Pythia | 1.4B | 10.8 | 10.2 | 11.3 | 15.2 |
+| **Mamba** | **1.4B** | **9.1** | **9.6** | **10.1** | **12.8** |
+| Pythia | 2.8B | 8.3 | 7.9 | 9.2 | 10.6 |
+| **Mamba** | **2.8B** | **7.4** | **7.2** | **8.3** | **9.1** |
+
+**Mamba consistently outperforms** Transformers of similar size by 10-20%
+
+### Zero-Shot Task Performance
+
+**Mamba-2.8B vs Transformer-2.7B** on common benchmarks:
+
+| Task | Mamba-2.8B | Transformer-2.7B | Delta |
+|------|------------|------------------|-------|
+| HellaSwag | 61.3 | 58.7 | +2.6 |
+| PIQA | 78.1 | 76.4 | +1.7 |
+| ARC-Easy | 68.2 | 65.9 | +2.3 |
+| ARC-Challenge | 42.7 | 40.1 | +2.6 |
+| WinoGrande | 64.8 | 62.3 | +2.5 |
+| OpenBookQA | 43.2 | 41.8 | +1.4 |
+| BoolQ | 71.4 | 68.2 | +3.2 |
+| MMLU (5-shot) | 35.2 | 33.8 | +1.4 |
+
+**Average improvement**: +2.2 points across benchmarks
+
+## Audio Modeling Benchmarks
+
+### SC09 (Speech Commands)
+
+**Task**: Audio classification (10 classes)
+
+| Model | Params | Accuracy | Inference (ms) |
+|-------|--------|----------|----------------|
+| Transformer | 8.2M | 96.2% | 18 ms |
+| S4 | 6.1M | 97.1% | 8 ms |
+| **Mamba** | **6.3M** | **98.4%** | **6 ms** |
+
+### LJSpeech (Speech Generation)
+
+**Task**: Text-to-speech quality (MOS score)
+
+| Model | Params | MOS ↑ | RTF ↓ |
+|-------|--------|-------|-------|
+| Transformer | 12M | 3.82 | 0.45 |
+| Conformer | 11M | 3.91 | 0.38 |
+| **Mamba** | **10M** | **4.03** | **0.21** |
+
+**RTF** (Real-Time Factor): Lower is better (0.21 = 5× faster than real-time)
+
+## Genomics Benchmarks
+
+### Human Reference Genome (HG38)
+
+**Task**: Next nucleotide prediction
+
+| Model | Context Length | Perplexity | Throughput |
+|-------|----------------|------------|------------|
+| Transformer | 1024 | 3.21 | 1,200 bp/s |
+| Hyena | 32768 | 2.87 | 8,500 bp/s |
+| **Mamba** | **1M** | **2.14** | **45,000 bp/s** |
+
+**Mamba handles million-length sequences** efficiently
+
+## Scaling Laws
+
+### Compute-Optimal Training
+
+**FLOPs vs perplexity** (The Pile validation):
+
+| Model Size | Training FLOPs | Mamba Perplexity | Transformer Perplexity |
+|------------|----------------|------------------|------------------------|
+| 130M | 6e19 | 28.1 | 29.6 |
+| 370M | 3e20 | 16.7 | 18.3 |
+| 790M | 8e20 | 12.3 | 13.9 |
+| 1.4B | 2e21 | 9.1 | 10.8 |
+| 2.8B | 6e21 | 7.4 | 8.3 |
+
+**Scaling coefficient**: Mamba achieves same perplexity as Transformer with **0.8×** compute
+
+### Parameter Efficiency
+
+**Perplexity 10.0 target** on The Pile:
+
+| Model Type | Parameters Needed | Memory (inference) |
+|------------|-------------------|-------------------|
+| Transformer | 1.6B | 3.2 GB |
+| **Mamba** | **1.1B** | **2.2 GB** |
+
+**Mamba needs ~30% fewer parameters** for same performance
+
+## Long-Range Arena (LRA)
+
+**Task**: Long-context understanding benchmarks
+
+| Task | Length | Transformer | S4 | Mamba |
+|------|--------|-------------|-----|-------|
+| ListOps | 2K | 36.4% | 59.6% | **61.2%** |
+| Text | 4K | 64.3% | 86.8% | **88.1%** |
+| Retrieval | 4K | 57.5% | 90.9% | **92.3%** |
+| Image | 1K | 42.4% | 88.7% | **89.4%** |
+| PathFinder | 1K | 71.4% | 86.1% | **87.8%** |
+| Path-X | 16K | OOM | 88.3% | **91.2%** |
+
+**Average**: Mamba 85.0%, S4 83.4%, Transformer 54.4%
+
+## Training Throughput
+
+### Tokens/sec During Training
+
+**8× A100 80GB** cluster, BF16, different sequence lengths:
+
+| Model | Seq Len 512 | Seq Len 2K | Seq Len 8K | Seq Len 32K |
+|-------|-------------|------------|------------|-------------|
+| Transformer-1.3B | 180K | 52K | OOM | OOM |
+| **Mamba-1.4B** | **195K** | **158K** | **121K** | **89K** |
+| Transformer-2.7B | 92K | 26K | OOM | OOM |
+| **Mamba-2.8B** | **98K** | **81K** | **62K** | **45K** |
+
+**Mamba scales to longer sequences** without OOM
+
+## Hardware Utilization
+
+### GPU Memory Bandwidth
+
+**Mamba-1.4B** inference on different GPUs:
+
+| GPU | Memory BW | Tokens/sec | Efficiency |
+|-----|-----------|------------|------------|
+| A100 80GB | 2.0 TB/s | 6,800 | 85% |
+| A100 40GB | 1.6 TB/s | 5,400 | 84% |
+| V100 32GB | 900 GB/s | 3,100 | 86% |
+| RTX 4090 | 1.0 TB/s | 3,600 | 90% |
+
+**High efficiency**: Mamba is memory-bandwidth bound (good!)
+
+### Multi-GPU Scaling
+
+**Mamba-2.8B** training throughput:
+
+| GPUs | Tokens/sec | Scaling Efficiency |
+|------|------------|-------------------|
+| 1× A100 | 12,300 | 100% |
+| 2× A100 | 23,800 | 97% |
+| 4× A100 | 46,100 | 94% |
+| 8× A100 | 89,400 | 91% |
+| 16× A100 | 172,000 | 88% |
+
+**Near-linear scaling** up to 16 GPUs
+
+## Cost Analysis
+
+### Training Cost (USD)
+
+**Training to The Pile perplexity 10.0** on cloud GPUs:
+
+| Model | Cloud GPUs | Hours | Cost (A100) | Cost (H100) |
+|-------|------------|-------|-------------|-------------|
+| Transformer-1.6B | 8× A100 | 280 | $8,400 | $4,200 |
+| **Mamba-1.1B** | **8× A100** | **180** | **$5,400** | **$2,700** |
+
+**Savings**: 36% cost reduction vs Transformer
+
+### Inference Cost (USD/million tokens)
+
+**API-style inference** (batch size 1, 2K context):
+
+| Model | Latency | Cost/M tokens | Quality (perplexity) |
+|-------|---------|---------------|---------------------|
+| Transformer-1.3B | 8.5 ms/tok | $0.42 | 10.8 |
+| **Mamba-1.4B** | **3.2 ms/tok** | **$0.18** | **9.1** |
+
+**Mamba provides**: 2.6× faster, 57% cheaper, better quality
+
+## Resources
+
+- Benchmarks code: https://github.com/state-spaces/mamba/tree/main/benchmarks
+- Paper (Mamba-1): https://arxiv.org/abs/2312.00752 (Section 4: Experiments)
+- Paper (Mamba-2): https://arxiv.org/abs/2405.21060 (Section 5: Experiments)
+- Pretrained models: https://huggingface.co/state-spaces