@tryhamster/gerbil 1.0.0-rc.8 → 1.0.0

package/docs/research/autoresearch-portable.md

@@ -0,0 +1,904 @@
+# Autoresearch: Complete Technical Reference
+
+A portable reference for implementing autonomous research loops. Based on [Karpathy's autoresearch](https://github.com/karpathy/autoresearch) and the [MLX port](https://github.com/trevin-creator/autoresearch-mlx).
+
+## Core Concept
+
+An AI agent autonomously iterates on a single mutable file, running fixed-time experiments, keeping improvements and reverting failures. Git provides memory. A single metric provides the keep/revert signal. Humans sleep; the agent works.
+
+Karpathy's first run: 83 experiments over ~2 days, 15 kept improvements, 11% speedup on the GPT-2 leaderboard. The agent found real improvements that a domain expert missed after years of manual tuning. All improvements were additive and transferred to larger models.
+
+The key insight is NOT that AI can tune hyperparameters — it's that the loop structure turns any measurable problem into an autonomous hill-climbing search where the agent builds on its own history, reads its own failures, and discovers things humans don't think to try.
+
+## Architecture
+
+```
+program.md          (immutable) — agent instructions, goals, constraints
+prepare.py          (immutable) — data, evaluation, constants
+constants.py        (immutable) — TIME_BUDGET, MAX_SEQ_LEN, EVAL_TOKENS
+train.py            (MUTABLE)   — the experiment subject
+results.tsv         (append)    — complete experiment log (all attempts)
+git branch          (ratcheted) — only successful commits survive
+run.log             (transient) — stdout/stderr from current experiment
+```
+
+### The Three-File Contract
+
+1. **program.md** — The agent's operating manual. Defines what to optimize, what rules to follow, what files are off-limits. The ONLY way humans influence the loop after launch. This is the "prompt" — the entire behavior of the system is determined by what's in this file. It has six sections: monorepo safety, setup protocol, experimentation rules, output format, logging rules, and the experiment loop.
+
+2. **prepare.py** — Fixed infrastructure. Data loading, tokenization, the evaluation function. Agent CANNOT modify this. This prevents metric gaming and ensures all experiments are comparable. It also contains the immutable constants (`TIME_BUDGET`, `MAX_SEQ_LEN`, `EVAL_TOKENS`). The evaluation function is the oracle — if the agent could change it, the system would be meaningless.
+
+3. **train.py** — The single mutable file. Agent can change anything: architecture, optimizer, hyperparameters, training loop, batch size, model size, activation functions, attention mechanisms, initialization schemes, learning rate schedules. The only constraint is it must run without crashing and produce output within the time budget.
+
+### System Parameters
+
+| Parameter | Value | Purpose |
+|-----------|-------|---------|
+| `MAX_SEQ_LEN` | 2048 tokens | Fixed sequence length for all experiments |
+| `TIME_BUDGET` | 300 seconds | Training duration (wall-clock, excludes init/eval) |
+| `EVAL_TOKENS` | 20,971,520 (H100) / 1,572,864 (MLX) | Validation evaluation budget |
+| `VOCAB_SIZE` | 8,192 | BPE tokenizer vocabulary |
+| `VAL_SHARD` | 6542 | Pinned validation shard (never trained on) |
+| `MAX_SHARD` | 6542 | Highest shard index |
+
+### Why This Works
+
+- **Fixed time budget** — every experiment costs the same wall-clock time (5 min default). This is the crucial design choice. It eliminates the explore/exploit tradeoff around compute allocation. The agent can try radical changes freely because the worst case is 5 wasted minutes. It also means the system naturally discovers the right tradeoff between model size and training steps — a bigger model gets fewer steps, a smaller one gets more.
+
+- **Single mutable file** — constrains search space, produces reviewable diffs, prevents the agent from accidentally breaking infrastructure. Every experiment is a single-file diff. You can `git log --stat` and see exactly what changed.
+
+- **Single metric** — val_bpb (validation bits per byte). Unambiguous: lower is better. No composite scores, no human judgment needed, no weighting decisions. The agent never has to ask "is this better?" — it just compares two numbers.
+
+- **Git as memory** — the commit log IS the research journal. The agent reads its own history to plan next experiments. Kept commits show what works; reverted commits (logged in results.tsv) show what doesn't. This dual-tracking is essential: the git branch is the clean history of validated improvements; results.tsv is the complete exploration log including all failures.
+
+- **Ratchet mechanism** — the branch only advances on improvement. You can never regress. The current HEAD is always the best-known configuration. This is a monotonic improvement guarantee that makes the system safe to run unattended.
+
+### Architectural Invariants
+
+These are the properties that make the system trustworthy:
+
+| Invariant | Enforcement | Rationale |
+|-----------|-------------|-----------|
+| Single mutable file | Agent instructions in program.md | Isolated experimental changes, reviewable diffs |
+| Fixed time budget | `TIME_BUDGET` constant in immutable prepare.py | Hardware-specific optimization, equal cost per experiment |
+| Fixed evaluation | `evaluate_bpb()` in immutable prepare.py | All experiments are comparable, no metric gaming |
+| No new dependencies | Agent instructions | Prevents scope creep, environment remains stable |
+| Single scalar metric | `val_bpb` only | Eliminates multi-objective complexity, unambiguous decisions |
+| Git-based ratcheting | Keep via commit, discard via reset | Monotonic improvement, clean history |
+| Immutable constants | `MAX_SEQ_LEN`, `VOCAB_SIZE`, `EVAL_TOKENS` | Consistent data processing across all experiments |
+| Pinned validation data | Shard 6542 never used for training | No data leakage, stable evaluation |
+
+## The Experiment Loop
+
+### Verbatim Protocol (from program.md)
+
+```
+LOOP FOREVER:
+1. Look at the git state: the current branch/commit we're on
+2. Tune train.py with an experimental idea by directly hacking the code
+3. git commit
+4. Run the experiment: uv run train.py > run.log 2>&1
+5. Read out the results: grep "^val_bpb:\|^peak_vram_mb:" run.log
+6. If grep output is empty, the run crashed. Read tail -n 50 run.log
+7. Record the results in results.tsv
+8. If val_bpb improved (lower), keep the git commit
+9. If val_bpb is equal or worse, git reset HEAD~1
+```
+
+### Exact Git Commands
+
+```bash
+# Setup (once, at start of run)
+git checkout -b autoresearch/<tag>          # e.g., autoresearch/mar5
+
+# Each experiment
+git add train.py
+git commit -m "experiment: <description>"
+uv run train.py > run.log 2>&1
+grep "^val_bpb:\|^peak_vram_mb:" run.log
+
+# If KEEP (val_bpb improved):
+git add results.tsv
+git commit --amend --no-edit                # fold results.tsv into experiment commit
+
+# If DISCARD (val_bpb equal or worse):
+git reset --hard <previous_kept_commit>     # revert to last known-good state
+```
+
+The amend-on-keep pattern is elegant: each kept commit contains both the code change AND the results that prove it worked. The branch reads as a clean research log.
+
+### Cycle Timing
+
+| Phase | H100 | Apple Silicon |
+|-------|------|---------------|
+| Training | 300s (5 min) | 300s (5 min) |
+| Model init + compilation | ~30s | ~11s |
+| Evaluation | ~30s | ~52s |
+| Agent analysis + code edit | ~60s | ~60s |
+| **Total per experiment** | ~7 min | ~7 min |
+| **Experiments per hour** | ~8 | ~8-9 |
+| **Overnight (10h)** | ~80 | ~80 |
+
+### The Setup Phase
+
+Before the autonomous loop begins, there's a critical interactive setup:
+
+1. **Propose a date-based run tag** (e.g., `mar5`, `jun12-m4max`)
+2. **Create the branch**: `git checkout -b autoresearch/<tag>`
+3. **Read all files**: README.md, prepare.py, train.py — build full context
+4. **Verify data**: Confirm `~/.cache/autoresearch/` contains shards and tokenizer
+5. **Create results.tsv** with header row
+6. **Run baseline**: Execute unmodified train.py, record as first entry
+7. **Get human approval** before starting autonomous loop
+
+**Critical**: The agent must establish its own baseline on the current hardware. A baseline from a different machine is invalid — the time budget produces different step counts on different hardware, which means different optimal configurations.
+
+## Decision Framework
+
+### The Primary Signal: val_bpb Comparison
+
+```
+IF val_bpb_new < val_bpb_best:
+    KEEP  — improvement detected
+ELIF val_bpb_new == val_bpb_best AND code_is_simpler:
+    KEEP  — simplification win
+ELSE:
+    DISCARD
+```
+
+### Concrete Decision Examples
+
+| Baseline val_bpb | New val_bpb | Code Change | Decision | Why |
+|-------------------|------------|-------------|----------|-----|
+| 0.997 | 0.993 | +5 clean lines | **Keep** | Clear improvement, reasonable complexity |
+| 0.997 | 0.996 | +20 hacky lines | Discard | Marginal gain, ugly complexity |
+| 0.997 | 0.997 | -15 lines (deletion) | **Keep** | Equal performance, simpler = win |
+| 0.997 | 0.992 | +10 clean lines | **Keep** | Significant gain, clean code |
+| 0.997 | 0.000 | Any | Crash | Log as crash, diagnose |
+| 0.997 | 1.005 | Any | Discard | Regression |
+| 0.997 | 0.996 | Removed entire subsystem | **Keep** | Great simplification for tiny cost |
+
+### The Simplicity Criterion
+
+"All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it."
+
+This is a soft constraint, not a hard rule. The agent applies judgment:
+- Code deletions achieving equal performance are explicit wins
+- Minor gains requiring significant complexity warrant rejection
+- Simplifications achieving equal performance are encouraged
+- Removing something and getting equal/better results is a great outcome — it means the thing was unnecessary
+
+### VRAM / Memory Management
+
+Memory is a soft constraint. The tradeoff is situational:
+
+| Memory Change | val_bpb Change | Decision |
+|---------------|----------------|----------|
+| +2 GB | -0.050 | **Keep** — meaningful gain |
+| +10 GB | -0.002 | Discard — not worth the memory |
+| -5 GB | +0.000 | **Keep** — efficiency win |
+| +1 GB | -0.010 | **Keep** — acceptable tradeoff |
+
+The principle: memory shouldn't "blow up dramatically." Some increase is fine for meaningful gains, but the agent shouldn't chase marginal improvements at the cost of 2x memory usage.
+
+## The Metric: val_bpb
+
+### What It Measures
+
+Validation bits per byte (val_bpb) is the primary optimization target. Lower = better.
+
+**Formula:**
+```
+val_bpb = total_nats / (ln(2) * total_bytes)
+```
+
+Where:
+- `total_nats` = sum of cross-entropy losses in natural log units
+- `total_bytes` = sum of UTF-8 byte lengths for all target tokens
+- `ln(2)` = conversion factor from nats to bits (≈0.6931)
+
+Special tokens (byte length 0) are excluded from the BPB calculation.
+
+### Why BPB Over Perplexity
+
+BPB is vocabulary-size-independent. If you change the tokenizer vocabulary (say from 8K to 32K tokens), perplexity becomes incomparable, but BPB remains valid because it normalizes by bytes, not tokens. This is critical for a system where the agent might want to experiment with different vocabulary sizes.
+
+### BPB Scale
+
+| BPB Value | Interpretation | Compression Ratio |
+|-----------|----------------|-------------------|
+| 8.0 | No compression (random) | 1:1 |
+| 2.0 | 4x compression | 4:1 |
+| 1.5 | 5.3x compression | 5.3:1 |
+| 1.0 | 8x compression (near SOTA) | 8:1 |
+
+Karpathy's H100 baseline: `val_bpb = 0.998` (8x compression in 5 min of training).
+Best MLX result: `val_bpb = 1.295` (6.2x compression on M4 Max in 5 min).
+
+### Evaluation Details
+
+```python
+def evaluate_bpb(model, tokenizer, batch_size):
+    """Fixed evaluation on pinned validation shard."""
+    steps = EVAL_TOKENS // (batch_size * MAX_SEQ_LEN)
+    total_nats = 0.0
+    total_bytes = 0
+
+    for inputs, targets in validation_batches:
+        # Forward pass with per-token loss (reduction='none')
+        per_token_loss = model(inputs, targets)  # cross-entropy, nats
+
+        # Look up byte count for each target token
+        token_byte_lengths = token_bytes[targets]
+
+        # Mask out special tokens (byte length 0)
+        mask = token_byte_lengths > 0
+
+        total_nats += (per_token_loss * mask).sum()
+        total_bytes += token_byte_lengths[mask].sum()
+
+    return total_nats / (math.log(2) * total_bytes)
+```
+
+Key properties:
+- Fixed validation shard (6542) — never used for training
+- Fixed token count — exactly 20,971,520 tokens (H100) or 1,572,864 (MLX)
+- Per-token loss with byte-length weighting — not averaged over tokens
+- Special token masking — tokens with 0 byte length excluded
+- Deterministic — same model always produces the same score
+
+## Key Rules
+
+### NEVER STOP
+
+The agent runs indefinitely until manually interrupted. No "should I keep going?" — the human might be asleep for 8+ hours. If the agent runs out of ideas, it should:
+
+1. Re-read train.py, prepare.py, and results.tsv
+2. Look at near-misses (experiments that almost improved)
+3. Try combining two near-miss ideas
+4. Try radical architectural changes
+5. Read papers (if the agent has access)
+6. Try the opposite of what's been working
+7. Try parameter sweeps in unexplored ranges
+8. Try removing things
+
+The design assumes overnight operation: 60-80 experiments without any human interaction.
+
+### Crash Handling Protocol
+
+```
+IF grep "^val_bpb:" run.log returns empty:
+    1. Read tail -n 50 run.log
+    2. Diagnose the failure
+    3. IF simple bug (typo, import, off-by-one):
+        Fix it, recommit, retry
+    4. IF fundamental issue (OOM, architectural impossibility):
+        Log as crash (val_bpb=0.000000, memory_gb=0.0)
+        Revert and move on
+    5. Record in results.tsv with status=crash
+```
+
+The agent uses judgment. A missing import is worth fixing. An idea that causes OOM on every variation is worth abandoning. The key is: don't get stuck. Log it, learn from it, move on.
+
+### Timeout Handling
+
+| Phase | Expected | Timeout Threshold | Action |
+|-------|----------|-------------------|--------|
+| Training | 300s | 600s (H100) / 900s (MLX) | Kill process, treat as failure |
+| Full cycle | ~7 min | 15 min | Kill, revert, log as crash |
+
+The training loop has a built-in fast-fail: if `train_loss > 100` at any point, exit immediately with code 1. This catches divergence early instead of wasting 5 minutes.
+
+### Results Logging
+
+`results.tsv` — tab-separated, NOT comma. Commas are explicitly prohibited in descriptions.
+
+```
+commit	val_bpb	memory_gb	status	description
+a1b2c3d	0.997900	44.0	keep	baseline
+b2c3d4e	0.993200	44.2	keep	increase LR to 0.04
+c3d4e5f	1.005000	44.0	discard	switch to GeLU activation
+d4e5f6g	0.000000	0.0	crash	double model width (OOM)
+e5f6g7h	0.995100	44.1	discard	add residual scaling (marginal + complex)
+```
+
+**Dual tracking**: results.tsv logs ALL experiments (keep/discard/crash). Git branch contains ONLY kept commits. This means you have both:
+- The **clean improvement history** (git log) — what the system converged to
+- The **full exploration log** (results.tsv) — what was tried, what failed, why
+
+The agent reads BOTH to plan next experiments. The failures are as informative as the successes.
+
+## Model Architecture (train.py)
+
+The mutable code starts as a GPT-2-style transformer. Everything below is the default starting point — the agent can change any of it.
+
+### GPTConfig
+
+```python
+@dataclass
+class GPTConfig:
+    vocab_size: int = 8192
+    max_seq_len: int = 2048      # matches MAX_SEQ_LEN in constants
+    n_layer: int                  # computed from DEPTH
+    n_head: int                   # computed from n_embd / HEAD_DIM
+    n_embd: int                   # computed from DEPTH * ASPECT_RATIO
+    head_dim: int = 64            # per-head dimension
+    window_pattern: str = "SSSL"  # attention pattern per layer
+```
+
+Model dimension is computed: `n_embd = ((DEPTH * ASPECT_RATIO + HEAD_DIM - 1) // HEAD_DIM) * HEAD_DIM` — rounded up to head_dim boundary.
+
+### Attention Mechanism
+
+- **Separate Q/K/V projections** (not fused)
+- **RoPE positional encoding** applied post-projection. Base theta defaults to 10K (agent discovered 100K is better)
+- **QK normalization** — queries and keys normalized before attention. Agent discovered post-norm scaling (`q, k *= 1.15`) helps for "sharper attention"
+- **Value Embeddings (VE)** — alternating layers add gated embeddings: `v = v + gate * ve` where `gate = 2 * sigmoid(linear(x))`. The gate channels and scale range are tunable
+- **Sliding window attention** — configurable per layer via `window_pattern` string. `"S"` = short-range (causal window), `"L"` = long-range (full context). Default `"SSSL"` = three short + one long. Agent discovered the window was too conservative
+- **Mask caching** by `(seq_len, window_size)` tuple to avoid recomputation
+- **Logit softcap**: `logits = cap * tanh(logits / cap)` where cap defaults to 20 (agent discovered 15 is better)
+
+### MLP
+
+- Configurable expansion factor (4x on H100, 3x on MLX — agent discovered 3x beats 4x)
+- **Squared ReLU** activation: `max(x, 0)^2`
+- No bias terms
+- Agent discovered initializing `c_fc` weights 0.5x smaller improves training
+
+### Block Structure
+
+```python
+x = x + attn(norm(x), ve, mask)   # pre-norm attention with value embeddings
+x = x + mlp(norm(x))              # pre-norm MLP
+```
+
+### Residual Interpolation (MLX variant)
+
+Learnable per-layer interpolation between residual stream and initial embedding:
+
+```python
+x = resid_lambdas[i] * x + x0_lambdas[i] * x0
+```
+
+Where `resid_lambdas` init to 1.0 and `x0_lambdas` init to 0.1. This allows direct paths from the input embedding to any layer, which helps with gradient flow in deeper models.
+
+### Weight Initialization
+
+| Parameter | Initialization | Scale |
+|-----------|----------------|-------|
+| Token embeddings | Normal(0, std) | std=1.0 (agent found 0.8 better) |
+| Output projection (lm_head) | Normal(0, 0.001) | Very small |
+| Q/K/V/MLP input weights | Uniform(-s, s) | s = sqrt(3) * n_embd^(-0.5) |
+| Projection output weights | Zero | Zero-init for residual |
+| resid_lambdas | 1.0 | Full residual |
+| x0_lambdas | 0.1 | Slight skip connection |
+| Value embeddings | Uniform(-s, s) | Same as Q/K/V |
+
+### Default Model Sizes
+
+| Config | DEPTH=8 (H100 default) | DEPTH=4 (MLX optimal) |
+|--------|------------------------|----------------------|
+| Parameters | ~50M | ~21M |
+| Steps in 5 min | ~46 | ~92 |
+| n_embd | ~768 | ~512 |
+
+The MLX agents universally discovered that DEPTH=4 beats DEPTH=8: half the parameters but 2x the training steps. **More optimizer steps beats more parameters when compute time is fixed.**
+
+### Output Format
+
+Training produces structured output parsed by the agent:
+
+```
+---
+val_bpb:          0.997900
+training_seconds: 300.1
+total_seconds:    325.9
+peak_vram_mb:     45060.2
+mfu_percent:      39.80
+total_tokens_M:   499.6
+num_steps:        953
+num_params_M:     50.3
+depth:            8
+```
+
+The agent extracts metrics via `grep "^val_bpb:\|^peak_vram_mb:" run.log`.
+
+## Optimizer System
+
+### Parameter Groups (4 groups with differentiated learning rates)
+
+The optimizer doesn't treat all parameters equally. This is one of the most important design choices:
+
+| Group | Which Parameters | Learning Rate | Weight Decay | Why Separate |
+|-------|-----------------|---------------|-------------|--------------|
+| **Matrix** | 2D weight matrices (not embed/unembed) | `MATRIX_LR` (0.04) | 0.2 | These are the bulk of the model; benefit from Muon |
+| **Embedding** | `tok_embed`, `embed_v` (value embeddings) | `EMBEDDING_LR` (0.6) | None | Embeddings need high LR, no decay |
+| **Unembedding** | `lm_head` | `UNEMBEDDING_LR` (0.004) | None | Output projection is sensitive |
+| **Scalar** | 1D parameters (norms, biases) | `SCALAR_LR` (0.5) | Custom | Small parameters, high LR |
+
+**Dimension scaling**: `dmodel_lr_scale = (model_dim / 768) ** -0.5` — learning rates scale inversely with model width.
+
+The agent discovered that per-group Adam betas and weight decay (instead of shared global values) was a significant improvement. Karpathy: "It found that AdamW betas were all messed up."
+
+### MuonAdamW Hybrid Optimizer
+
+Two update strategies in one optimizer:
+
+**AdamW** (for embeddings, unembedding, scalars):
+```python
+# Standard Adam with decoupled weight decay
+m = beta1 * m + (1 - beta1) * grad                    # first moment
+v = beta2 * v + (1 - beta2) * grad^2                  # second moment
+m_hat = m / (1 - beta1^t)                              # bias correction
+v_hat = v / (1 - beta2^t)
+param = param * (1 - lr * weight_decay)                # decoupled decay
+param = param - lr * m_hat / (sqrt(v_hat) + eps)       # Adam step
+```
+
+Default AdamW hyperparameters:
+| Parameter | Default |
+|-----------|---------|
+| beta1 | 0.8 |
+| beta2 | 0.95 |
+| weight_decay | 0.2 |
+| eps | 1e-8 |
+
+**Muon** (for weight matrices — the bulk of the model):
+
+A second-order optimizer using Newton-Schulz iterations to precondition gradients. This is what makes the H100 version fast — Muon gets more signal per gradient step than AdamW alone.
+
+```python
+# Newton-Schulz preconditioning
+G = gradient
+for i in range(NS_STEPS):
+    G = G @ (3*I - G.T @ G) / 2     # iterative polar decomposition
+# Apply preconditioned gradient with momentum
+```
+
+The Newton-Schulz iteration finds the "best direction" to update weights by whitening the gradient — removing correlations so each parameter gets an equally-scaled update. It's like applying a second-order method (Newton's method) but cheaply.
+
+Key Muon parameters:
+| Parameter | H100 Default | MLX Optimal | Notes |
+|-----------|-------------|-------------|-------|
+| NS_STEPS | 5 | **3** | Fewer iterations = faster steps = more updates in 5 min |
+| momentum | 0.95 | 0.95 | Baseline momentum |
+| beta2 | 0.95 | — | Agent found 0.9 better on H100 |
+| momentum_warmup | 0.95→0.97 | — | Over 400 steps on H100 |
+
+**MLX discovery**: NS_STEPS=3 outperforms NS_STEPS=5 on Apple Silicon. This is the first documented tuning of Muon on this hardware. The reasoning: fewer iterations per step = faster steps = more total gradient updates in the fixed 5-minute budget.
+
+**Hardware-dependent optimizer choice**: On the Mac Mini (constrained compute), Muon was a breakthrough. On M4 Max (more headroom), plain AdamW won. The system discovers hardware-appropriate configurations, not a single "best" configuration.
+
+### Learning Rate Schedule
+
+Three phases controlled by `progress = total_training_time / TIME_BUDGET`:
+
+```
+Phase 1: Warmup       [0, WARMUP_RATIO]        → linear ramp from 0 to 1.0
+Phase 2: Steady state [WARMUP_RATIO, 1-WARMDOWN_RATIO] → constant at 1.0
+Phase 3: Warmdown     [1-WARMDOWN_RATIO, 1.0]  → linear decay to FINAL_LR_FRAC
+```
+
+| Parameter | H100 Default | H100 Optimized | MLX Default |
+|-----------|-------------|----------------|-------------|
+| WARMUP_RATIO | ratio-based | 40 absolute steps | 0.0 |
+| WARMDOWN_RATIO | 0.5 | 0.65 | 0.5 |
+| FINAL_LR_FRAC | 0.0 | 0.05 | 0.0 |
+
+The agent discovered:
+- **Non-zero FINAL_LR_FRAC (0.05)** — don't decay LR all the way to zero. Keep a small residual learning rate
+- **Longer warmdown (0.65 vs 0.5)** — more gradual cooldown helps
+- **Weight decay schedule**: linear → cosine decay was an improvement
+
+## Training Loop Internals
+
+### Time Budget Enforcement
+
+```python
+TIME_BUDGET = 300  # seconds of actual training
+STARTUP_EXCLUDE_STEPS = 1  # exclude first step from timing (compilation overhead)
+
+t0 = None
+for step in range(max_steps):
+    # ... forward, backward, optimizer step ...
+
+    if step == STARTUP_EXCLUDE_STEPS:
+        t0 = time.perf_counter()  # start timing AFTER compilation
+
+    if t0 is not None:
+        total_training_time = time.perf_counter() - t0
+        if step >= STARTUP_EXCLUDE_STEPS and total_training_time >= TIME_BUDGET:
+            break
+```
+
+Wall-clock measurement EXCLUDES: model initialization, first-step compilation, data loading setup, final evaluation.
+Wall-clock measurement INCLUDES: forward passes, backward passes, optimizer steps, gradient accumulation.
+
+This means every experiment gets exactly the same amount of actual training compute, regardless of initialization overhead.
+
+### Progress-Based Scheduling
+
+The LR schedule is driven by wall-clock progress, not step count:
+
+```python
+progress = min(total_training_time / TIME_BUDGET, 1.0)
+lr_multiplier = get_lr_multiplier(progress)
+```
+
+This is important because different configurations produce different step counts in the same 5 minutes. A model with 2x parameters takes 2x longer per step, so it gets half the steps — but the LR schedule still spans the full training run proportionally.
+
+### Fast-Fail Detection
+
+```python
+if train_loss > 100:
+    sys.exit(1)  # divergence detected, abort immediately
+```
+
+Don't waste 5 minutes on a diverged run. If loss explodes, bail immediately.
+
+### Smoothed Loss Tracking
+
+```python
+smoothed_loss = beta * smoothed_loss + (1 - beta) * loss  # EMA with beta=0.9
+smoothed_loss_debiased = smoothed_loss / (1 - beta^step)  # early-step correction
+```
+
+### Memory Management (MLX-specific)
+
+```python
+gc.collect()
+gc.freeze()    # freeze all existing objects (exclude from GC)
+gc.disable()   # disable GC during training
+
+# Every 5000 steps:
+gc.collect()   # manual collection to prevent memory drift
+```
+
+Aggressive GC management is critical on unified memory hardware where training and system share the same pool.
+
+### Batch Size Configuration
+
+| Parameter | H100 Default | MLX Default | MLX Optimal |
+|-----------|-------------|-------------|-------------|
+| TOTAL_BATCH_SIZE | 2^17 | 2^16 (65,536 tokens) | 2^14 (16,384 tokens) |
+| DEVICE_BATCH_SIZE | — | 16 sequences | — |
+| GRAD_ACCUM_STEPS | — | computed | — |
+
+```python
+grad_accum_steps = TOTAL_BATCH_SIZE // (DEVICE_BATCH_SIZE * MAX_SEQ_LEN)
+```
+
+The agents discovered smaller batch sizes outperform: `TOTAL_BATCH_SIZE 2^14-2^13` beat `2^17` by fitting more gradient steps in the time budget. This is the same insight as DEPTH=4 vs DEPTH=8: **in a fixed time budget, more steps > more tokens per step.**
+
+## Data Pipeline (prepare.py — Immutable)
+
+### Dataset
+
+`karpathy/climbmix-400b-shuffle` from HuggingFace:
+- 6,543 parquet shards total
+- Training: shards 0-6,541
+- Validation: shard 6,542 (pinned, never used for training)
+
+### Download System
+
+```python
+def download_data():
+    """Parallel download with retry logic."""
+    pool = multiprocessing.Pool(processes=8)
+    # For each shard:
+    #   1. Download to .tmp file
+    #   2. Atomic rename on success
+    #   3. Skip if already exists (resumable)
+    #   4. Retry with exponential backoff (3-5 attempts, wait 2^attempt seconds)
+```
+
+Key properties:
+- **Atomic writes**: download to `.tmp`, rename on completion — prevents corruption from interrupted downloads
+- **Resumable**: skips already-downloaded files
+- **Parallel**: 8 workers for throughput
+- **Cached**: `~/.cache/autoresearch/data/`
+
+### Tokenizer
+
+- **BPE** via `rustbpe` library (Rust-based, fast)
+- **Vocabulary size**: 8,192 (minus 4 special tokens = 8,188 mergeable ranks)
+- Trained on ~1 billion characters from training shards
+- Integrated with `tiktoken` for encoding/decoding
+- Produces: `tokenizer.pkl` (tiktoken Encoding) and `token_bytes.npy`/`token_bytes.pt` (maps token IDs → UTF-8 byte lengths)
+- Special tokens have byte length 0 (excluded from BPB)
+
+### BOS-Aligned Best-Fit Packing
+
+The dataloader achieves 100% token utilization with no padding:
+
+```python
+def make_dataloader(tokenizer, batch_size, seq_len, split, buffer_size=1000):
+    """
+    1. Tokenize documents, prepend BOS to each
+    2. Buffer 1000 tokenized documents
+    3. Best-fit selection: pack documents into rows of exactly seq_len+1 tokens
+    4. Yield (inputs, targets, epoch) where:
+       - inputs = positions [0, seq_len)
+       - targets = positions [1, seq_len+1)
+    """
+```
+
+Properties:
+- BOS token prepended to each document — every document boundary is marked
+- Best-fit selection minimizes wasted tokens (unlike fixed-length chunking)
+- Deterministic: same shard order produces identical batches
+- Row group processing: reads parquet by row group for memory efficiency
+- Infinite iterator with epoch tracking
+
+### Cache Structure
+
+```
+~/.cache/autoresearch/
+├── data/
+│   ├── shard_00000.parquet      (training)
+│   ├── shard_00001.parquet
+│   ├── ...
+│   ├── shard_06541.parquet      (training)
+│   └── shard_06542.parquet      (validation — pinned)
+└── tokenizer/
+    ├── tokenizer.pkl            (tiktoken Encoding object)
+    └── token_bytes.npy/.pt      (vocab_size x int32, byte lengths)
+```
+
+One-time setup: `uv run prepare.py` (~5 minutes, downloads data and trains tokenizer).
+
+## Agent Prompt Engineering (program.md)
+
+The program.md is the most important file in the system. It's what turns a general-purpose AI agent into an autonomous researcher. Key design choices:
+
+### Structure (6 sections)
+
+1. **Monorepo Safety** — If in a monorepo, stage only the experiment directory paths. Never `git add -A`.
+
+2. **Setup Protocol** — Interactive initialization: branch creation, file reading, data verification, baseline establishment, human approval. This prevents the agent from running blind.
+
+3. **Experimentation Rules** — Hard constraints: only modify train.py, no new dependencies, no modifying prepare.py or constants, no changing evaluation function or time/sequence length constants.
+
+4. **Output Format** — What to grep for. The structured `---` block with `val_bpb:`, `peak_vram_mb:`, etc.
+
+5. **Logging Rules** — TSV format, no commas, status values (keep/discard/crash), crash logging convention (`val_bpb=0.000000`).
+
+6. **The Experiment Loop** — The autonomous cycle, verbatim.
+
+### Key Prompt Engineering Decisions
+
+**The NEVER STOP principle** is repeated and emphasized. This is deliberate — without it, agents naturally pause to ask for confirmation, which defeats overnight operation.
+
+**The simplicity criterion** is stated as a tiebreaker, not a primary objective. This prevents the agent from refusing to add code — it can add complexity if the improvement justifies it.
+
+**The crash handling** is judgment-based ("fix simple bugs, skip fundamentally broken ideas"). This avoids both extremes: an agent that gives up at the first error, or an agent that retries the same broken idea forever.
+
+**The idea exhaustion strategy** ("re-read files, try combining near-misses, try radical changes") prevents the agent from stalling when it runs out of obvious ideas. The instruction to "think harder" is deliberate — agents can often find ideas if they're told not to give up.
+
+## Real Results
+
+### Karpathy's 2-Day Run (H100)
+
+83 experiments, 15 kept improvements. Baseline: `val_bpb = 0.998`, 45.1 GB VRAM.
+
+**Optimizer & schedule changes:**
+- Unembedding LR: 0.004 → 0.008, weight decay: 0.2 → 0.28
+- Per-group Adam betas and weight decay (instead of shared global)
+- Muon beta2: 0.95 → 0.9, momentum warmup target: 0.95 → 0.97 over 400 steps
+- Warmup: ratio-based → absolute steps (40)
+- Warmdown ratio: 0.5 → 0.65, final LR fraction: 0.0 → 0.05
+- Weight decay schedule: linear → cosine decay
+- Polar express norm factor: 1.02 → 1.01
+
+**Architecture & init changes:**
+- VE gate: channels 32 → 12, scale range 2x → 3x, init small positive
+- Post-QK-norm scaling (q,k *= 1.15) for sharper attention
+- Embedding init std: 1.0 → 0.8, MLP c_fc init 0.5x smaller
+- RoPE base theta: 10K → 100K
+- Short attention window: seq_len/2 → ~seq_len/3 (ceil to 128 tile)
+- Logit softcap: 20 → 15
+
+Result: "Time to GPT-2" dropped from 2.02 hours to 1.80 hours (11% improvement).
+
+Key quote: "The agent found multipliers to sharpen attention, pointing to future work. It found that Value Embeddings really like regularization and I wasn't applying any (oops). It found that my banded attention was too conservative (I forgot to tune it). It found that AdamW betas were all messed up."
+
+What this means: the agent found bugs and missed tuning opportunities in code written by one of the world's foremost ML researchers. The improvements were real, not artifacts — they transferred to larger models and stacked additively.
+
+### MLX Port Overnight Results (Apple Silicon)
+
+Three machines ran autonomously for 6-12 hours:
+
+| Machine | Optimizer | Experiments | Best val_bpb | Improvement |
+|---|---|---|---|---|
+| M4 Max 128GB | AdamW | ~50 | 1.295 | 19% |
+| M4 Max 128GB (#2) | AdamW + surface gates | ~30 | 1.339 | 17% |
+| Mac Mini | Muon + AdamW | 30 | 1.462 | 24% |
+
+Upstream H100 reference: val_bpb 0.998 in the same 5-minute budget.
+
+### Universal Discoveries (all machines converged)
+
+- **DEPTH=4 over DEPTH=8**: Half the parameters, 2x training steps. Every machine found this independently — "more optimizer steps beats more parameters when compute time is fixed"
+- **Smaller batch sizes**: 2^14-2^13 beat 2^17 — more gradient updates matter more than more tokens per update
+- **Lean MLP**: 3x expansion beat 4x. On Mac Mini (most constrained), 2x was better
+- **Schedule tuning**: WARMDOWN_RATIO and FINAL_LR_FRAC were significant everywhere
+
+### Hardware-Specific Discoveries
+
+- **Muon is hardware-dependent**: breakthrough on Mac Mini (constrained compute), but plain AdamW won on M4 Max. The hypothesis: when you have plenty of memory/compute, AdamW's simplicity wins; when compute is tight, Muon's better gradient signal per step matters more
+- **NS_STEPS=3 over NS_STEPS=5**: First documented Muon tuning on Apple Silicon. Fewer Newton-Schulz iterations = faster steps = more total updates
+- **Same loop + different hardware = genuinely different optimal configurations**. That's the point — the system finds what's best for YOUR hardware, not a universal recipe
+
+### Anti-Patterns Discovered
+
+These consistently failed across machines:
+
+1. **Increasing model size beyond optimal depth** — fewer training steps in fixed budget, net negative
+2. **Large batch sizes (2^17+)** — fewer gradient updates, optimizer progress stalls
+3. **Complex architectural changes with tiny gains** — failed simplicity criterion
+4. **Over-expanding MLP (4x+)** — computation cost not worth the extra capacity
+5. **Any change that reduces step count significantly** — the time budget makes step count critical
+
+### Common Successful Parameter Ranges
+
+These are the ranges where agents found improvements across runs:
+
+| Parameter | Explored Range | Typical Optimal |
+|-----------|---------------|-----------------|
+| DEPTH | 4-8 | 4 (universal on MLX) |
+| WINDOW_PATTERN | "SL", "SSSL", "SSSSL" | "SSSL" |
+| MLP expansion | 2x-4x | 3x (2x on constrained hw) |
+| HEAD_DIM | 64-192 | 64-128 |
+| TOTAL_BATCH_SIZE | 2^13-2^17 | 2^14 |
+| MATRIX_LR | 0.01-0.1 | 0.04 |
+| EMBEDDING_LR | 0.3-1.2 | 0.6 |
+| WARMUP_RATIO | 0.0-0.1 | 0.0 |
+| WARMDOWN_RATIO | 0.2-0.5 | 0.3-0.5 |
+
+## Monitoring
+
+### During a Run
+
+Real-time progress in `run.log`:
+```
+step    1 | loss 11.2345 | lr 1.2e-04 | 2.3k tokens/s
+step    2 | loss 10.8734 | lr 2.4e-04 | 2.4k tokens/s
+...
+step   92 | loss  2.1234 | lr 3.8e-05 | 2.3k tokens/s
+---
+val_bpb:          1.534000
+training_seconds: 312.4
+total_seconds:    405.7
+peak_vram_mb:     27528.9
+mfu_percent:      0.00
+total_tokens_M:   39.8
+num_steps:        92
+num_params_M:     21.3
+depth:            4
+```
+
+### Simple Monitoring Script
+
+```bash
+# Watch results accumulate
+while true; do
+  echo "--- $(date) ---"
+  tail -5 results.tsv | column -t -s $'\t'
+  echo "Total: $(wc -l < results.tsv) experiments"
+  sleep 60
+done
+```
+
+### Multi-Machine Runs
+
+Branch naming convention: `autoresearch/<date>-<machine>`
+
+```bash
+# Machine 1
+git checkout -b autoresearch/mar5-m4max
+
+# Machine 2
+git checkout -b autoresearch/mar5-mini
+```
+
+After overnight runs, compare branches. Both machines will discover universal improvements (DEPTH=4) and hardware-specific ones (optimizer choice). Cross-pollinate: try Machine 1's best config on Machine 2 and vice versa.
+
+## Adapting to Other Domains
+
+The pattern generalizes to any optimization problem with:
+1. A mutable configuration/code (the "train.py")
+2. An objective metric that's efficient to evaluate
+3. A fixed budget per experiment
+4. A keep/revert mechanism (git)
+
+### Template
+
+```yaml
+# autoresearch-config.yaml
+name: "my-project"
+metric: "the_metric_name"          # what to optimize
+metric_direction: "minimize"        # or "maximize"
+
+mutable_files:
+  - "the_file_agent_can_edit.py"
+
+immutable_files:
+  - "evaluation.py"                 # metric computation, cannot be gamed
+  - "data_loader.py"                # fixed data pipeline
+
+run_command: "python train.py > run.log 2>&1"
+eval_command: "grep '^metric:' run.log"
+
+budget_minutes: 5                   # fixed time per experiment
+branch_prefix: "autoresearch"       # git branch naming
+
+rules:
+  - "Only modify files listed in mutable_files"
+  - "Do not install new dependencies"
+  - "Simpler solutions preferred over complex ones"
+  - "Run indefinitely until interrupted"
+```
+
+### Example Domains
+
+**ML model training** (the original):
+- Mutable: train.py (architecture, optimizer, hyperparams)
+- Metric: val_bpb or val_loss
+- Budget: 5 min per experiment
+
+**Inference optimization** (like Gerbil's kernel optimizer):
+- Mutable: config.toml, shader code, kernel parameters
+- Metric: tokens/second, latency_p99
+- Budget: 2 min per benchmark run
+
+**Compiler/codegen optimization**:
+- Mutable: optimization passes, code generation rules
+- Metric: benchmark suite runtime
+- Budget: 10 min (compile + bench)
+
+**Growth/marketing**:
+- Mutable: landing page copy, ad targeting config
+- Metric: conversion rate
+- Budget: hours (need traffic for statistical significance)
+
+**Fine-tuning pipeline**:
+- Mutable: training config (hyperparams, data mix, LoRA settings)
+- Metric: composite eval score (pass rate + preference rate)
+- Budget: 30-90 min per cloud training run
+
+### Key Considerations for Adaptation
+
+1. **Cycle time is king**. Karpathy gets ~80 experiments overnight because each takes ~7 minutes total. If your cycle is 90 minutes, you get ~16/day. Find proxy metrics that correlate with your true objective but evaluate faster.
+
+2. **The metric must be automatable**. Human judgment doesn't scale. Either automate evaluation entirely (val_bpb) or use an AI judge (Opus scoring responses). The metric must be a single scalar that the agent can compare with `<`.
+
+3. **The mutable surface should be small**. One file, or a small set of config values. If the agent can change everything, the search space explodes and improvements don't stack reliably. One-file diffs are reviewable; multi-file changes are opaque.
+
+4. **Git ratchet prevents regression**. This is critical. You never go backwards. Every kept commit is guaranteed to be at least as good as the previous best. This is what makes overnight operation safe.
+
+5. **The agent needs memory**. results.tsv + git log give the agent context on what worked and what didn't. Without this, the agent repeats failed experiments. The dual tracking (git for successes, TSV for everything) is essential.
+
+6. **Establish your own baseline**. Never use someone else's baseline numbers. Run the unmodified code on your hardware and measure. Different hardware, different step counts, different optimal configurations.
+
+7. **The time budget creates natural tradeoffs**. You don't need to manually balance model size vs. training steps — the fixed budget does it automatically. A bigger model gets fewer steps; the metric tells you which tradeoff wins.
+
+8. **Hardware-specific optimization is a feature, not a bug**. The same loop on different hardware discovers different optimal configurations. This is correct behavior — the best config for an H100 is not the best config for a Mac Mini.
+
+## The Vision
+
+From Karpathy:
+
+> "All LLM frontier labs will do this. It's the final boss battle. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges."
+
+> "Any metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm."
+
+> "One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ritual of 'group meeting'. That era is long gone."
+
+## References
+
+- [karpathy/autoresearch](https://github.com/karpathy/autoresearch) — original repo
+- [karpathy/nanochat](https://github.com/karpathy/nanochat) — the training codebase being optimized
+- [nanochat commit 6ed7d1d](https://github.com/karpathy/nanochat/commit/6ed7d1d82cee16c2e26f45d559ad3338447a6c1b) — the stacked improvements from round 1
+- [trevin-creator/autoresearch-mlx](https://github.com/trevin-creator/autoresearch-mlx) — Apple Silicon port
+- [DeepWiki: autoresearch](https://deepwiki.com/karpathy/autoresearch) — detailed system documentation
+- [DeepWiki: autoresearch-mlx](https://deepwiki.com/trevin-creator/autoresearch-mlx) — MLX port documentation