nanogpt 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c2d0853148f473bb23f4ffffaa5a7b9e45f3faff570cc0a85253d40b6b63b80a
-  data.tar.gz: 68f31e52460273d97d1a97de844fb79c9bfab22b939a897482b6def4c77bdecc
+  metadata.gz: f3563f92bd2b986ae27476c80c596cca77438d899b53d017f6cb708179fd7ad4
+  data.tar.gz: d15ed5effb810c0fa022b4fa3d6708fc2623332b0d3f4de5057a66343a9a3b2f
 SHA512:
-  metadata.gz: 42a617a95e04d7727cc011a033ea039b8763a9103093dbec7ad614eb74a9a4f4ed818d82ffedeb8e627494dd73fd7950fdb30f97dd5ff63866123a0e892befc1
-  data.tar.gz: af2916e7179830cc91fc516be16b765e46a3c6f37cb732583900bf71591abf16d8bbdc250bbe07a0b83eac124a5ec5b7241684017b4c166ebe3fcb8611702592
+  metadata.gz: 030bd007f053f6400afec46810e10d1c8a96b881df932c291fc7efb637a4c00424cd9304facac71e56783a861417dcd6ec247b2f6df3926a443d2540ecf6adcf
+  data.tar.gz: c0bcf4eae610c2475c841c186ee8a2a6aebc2bb22ecf2d0fbb9d65c7218152699598d320cc6972262ff918d0991c4cc4c951a68552bec46b24bd4fee10aa1b07

data/Gemfile.lock

@@ -1,17 +1,35 @@
 PATH
   remote: .
   specs:
-    nanogpt (0.1.2)
+    nanogpt (0.3.0)
       numo-narray (~> 0.9)
+      rackup (~> 2.0)
+      sinatra (~> 4.0)
+      sqlite3 (~> 2.0)
       tiktoken_ruby (~> 0.0)
       torch-rb (~> 0.14)
+      webrick (~> 1.8)
 
 GEM
   remote: https://rubygems.org/
   specs:
+    base64 (0.3.0)
     diff-lcs (1.6.2)
+    logger (1.7.0)
+    mustermann (3.0.4)
+      ruby2_keywords (~> 0.0.1)
     numo-narray (0.9.2.1)
     parquet (0.7.3-arm64-darwin)
+    rack (3.2.5)
+    rack-protection (4.2.1)
+      base64 (>= 0.1.0)
+      logger (>= 1.6.0)
+      rack (>= 3.0.0, < 4)
+    rack-session (2.1.1)
+      base64 (>= 0.1.0)
+      rack (>= 3.0.0)
+    rackup (2.3.1)
+      rack (>= 3)
     rice (4.7.1)
     rspec (3.13.2)
       rspec-core (~> 3.13.0)
@@ -26,9 +44,20 @@ GEM
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.13.0)
     rspec-support (3.13.6)
+    ruby2_keywords (0.0.5)
+    sinatra (4.2.1)
+      logger (>= 1.6.0)
+      mustermann (~> 3.0)
+      rack (>= 3.0.0, < 4)
+      rack-protection (= 4.2.1)
+      rack-session (>= 2.0.0, < 3)
+      tilt (~> 2.0)
+    sqlite3 (2.9.0-arm64-darwin)
     tiktoken_ruby (0.0.13-arm64-darwin)
+    tilt (2.7.0)
     torch-rb (0.22.2)
       rice (>= 4.7)
+    webrick (1.9.2)
 
 PLATFORMS
   arm64-darwin-24

data/docs/ARCHITECTURE.md

@@ -0,0 +1,429 @@
+# nanoGPT Architecture
+
+This document explains the architecture of nanoGPT, a Ruby implementation of GPT-2 style language models.
+
+## Overview
+
+nanoGPT implements a **decoder-only transformer** architecture for autoregressive language modeling. The model learns to predict the next token in a sequence given all previous tokens.
+
+```
+Input Tokens → Embeddings → Transformer Blocks → Output Logits → Next Token Prediction
+```
+
+## Core Components
+
+### 1. Token & Position Embeddings
+
+The model converts discrete tokens into continuous vectors:
+
+```
+Token IDs [0, 42, 7, ...] → Token Embeddings (vocab_size × n_embd)
+Position [0, 1, 2, ...]   → Position Embeddings (block_size × n_embd)
+
+Final Embedding = TokenEmbed(token) + PosEmbed(position)
+```
+
+**Files:** `lib/nano_gpt/model.rb:16-18`
+
+The position embeddings are learned (not sinusoidal) and allow the model to understand token order.
+
+### 2. Transformer Block
+
+Each transformer block contains two sub-layers with residual connections:
+
+```
+┌─────────────────────────────────────────────────┐
+│                Transformer Block                │
+├─────────────────────────────────────────────────┤
+│                                                 │
+│   x ─→ LayerNorm ─→ Attention ─→ (+) ─┐        │
+│   └────────────────────────────────────┘        │
+│                                                 │
+│   x ─→ LayerNorm ─→ MLP ─→ (+) ─→ output       │
+│   └─────────────────────────────────────┘       │
+│                                                 │
+└─────────────────────────────────────────────────┘
+```
+
+**File:** `lib/nano_gpt/layers/block.rb`
+
+This is "pre-norm" architecture (LayerNorm before sub-layer, not after) which improves training stability.
+
+### 3. Causal Self-Attention
+
+The attention mechanism allows each token to attend to all previous tokens (but not future ones):
+
+```
+Q = x @ W_q    (Query: what am I looking for?)
+K = x @ W_k    (Key: what do I contain?)
+V = x @ W_v    (Value: what information do I provide?)
+
+Attention(Q,K,V) = softmax(Q @ K^T / sqrt(d_k)) @ V
+```
+
+The "causal" part means we mask future positions:
+
+```
+Position:    0   1   2   3
+Token 0:    [1   0   0   0]  ← can only see itself
+Token 1:    [1   1   0   0]  ← can see positions 0,1
+Token 2:    [1   1   1   0]  ← can see positions 0,1,2
+Token 3:    [1   1   1   1]  ← can see all
+```
+
+**File:** `lib/nano_gpt/layers/causal_self_attention.rb`
+
+Multi-head attention splits the embedding into `n_head` parallel attention operations, allowing the model to attend to different aspects simultaneously.
+
+### 4. MLP (Feed-Forward Network)
+
+After attention, each token passes through an MLP:
+
+```
+x → Linear(n_embd → 4×n_embd) → GELU → Linear(4×n_embd → n_embd) → output
+```
+
+**File:** `lib/nano_gpt/layers/mlp.rb`
+
+The 4× expansion provides additional capacity for computation.
+
+### 5. Output Layer (Weight Tying)
+
+The final output uses the transposed token embedding matrix:
+
+```
+logits = LayerNorm(x) @ TokenEmbed.T
+```
+
+This "weight tying" reduces parameters and improves performance.
+
+## Data Flow
+
+```
+┌────────────────────────────────────────────────────────────────────┐
+│                        GPT Forward Pass                            │
+├────────────────────────────────────────────────────────────────────┤
+│                                                                    │
+│  Input: token_ids [batch, seq_len]                                 │
+│           ↓                                                        │
+│  Token Embedding + Position Embedding                              │
+│           ↓                                                        │
+│  Dropout                                                           │
+│           ↓                                                        │
+│  ┌──────────────────────────────────────┐                          │
+│  │ Transformer Block 1                  │                          │
+│  │   LayerNorm → Attention → Residual   │                          │
+│  │   LayerNorm → MLP → Residual         │                          │
+│  └──────────────────────────────────────┘                          │
+│           ↓                                                        │
+│  ... (repeat n_layer times) ...                                    │
+│           ↓                                                        │
+│  Final LayerNorm                                                   │
+│           ↓                                                        │
+│  Linear projection (tied weights)                                  │
+│           ↓                                                        │
+│  Output: logits [batch, seq_len, vocab_size]                       │
+│                                                                    │
+└────────────────────────────────────────────────────────────────────┘
+```
+
+## Training Pipeline
+
+```
+┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
+│   Prepare   │ →  │    Load     │ →  │   Forward   │ →  │  Backward   │
+│    Data     │    │   Batch     │    │    Pass     │    │    Pass     │
+└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
+      │                  │                  │                  │
+   train.bin        DataLoader           GPT.forward      loss.backward
+   val.bin          get_batch()           ↓                   ↓
+   meta.json           ↓               Cross-entropy      Gradients
+                   [x, y] tensors         loss             computed
+                                                              │
+                                                              ↓
+┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
+│   Update    │ ←  │    Clip     │ ←  │  Accumulate │ ←  │             │
+│   Weights   │    │  Gradients  │    │  Gradients  │    │             │
+└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
+      │
+   AdamW
+   optimizer
+```
+
+**File:** `lib/nano_gpt/trainer.rb`
+
+## Generation (Inference)
+
+```
+Start: "Hello"
+        ↓
+Tokenize: [15496]
+        ↓
+┌─────────────────────────────────────┐
+│ Loop until max_tokens:              │
+│   1. Forward pass → logits          │
+│   2. Apply temperature              │
+│   3. Apply top-k filtering          │
+│   4. Sample from distribution       │
+│   5. Append to sequence             │
+└─────────────────────────────────────┘
+        ↓
+Decode: "Hello, how are you today?"
+```
+
+**File:** `lib/nano_gpt/model.rb:110-147`
+
+## File Structure
+
+```
+lib/nano_gpt/
+├── model.rb              # GPT model class
+├── config.rb             # GPTConfig (model hyperparameters)
+├── train_config.rb       # Training/sampling/bench configs
+├── trainer.rb            # Training loop
+├── data_loader.rb        # Batch loading from binary files
+├── tokenizer.rb          # Character & GPT-2 BPE tokenizers
+├── lr_scheduler.rb       # Cosine annealing with warmup
+├── device.rb             # CPU/CUDA/MPS device detection
+├── textfile_preparer.rb  # Custom dataset preparation
+└── layers/
+    ├── block.rb                  # Transformer block
+    ├── causal_self_attention.rb  # Multi-head attention
+    ├── mlp.rb                    # Feed-forward network
+    └── layer_norm.rb             # Layer normalization
+```
+
+---
+
+# Glossary
+
+## A
+
+### Attention
+A mechanism that allows each position in a sequence to dynamically focus on other positions. Computes weighted sums of values based on query-key similarities.
+
+### Autoregressive
+A model that generates outputs one step at a time, where each step depends on all previous steps. GPT generates text left-to-right, predicting one token at a time.
+
+### AdamW
+An optimizer that combines Adam (adaptive learning rates) with decoupled weight decay. Used for training the model.
+
+## B
+
+### Batch Size
+The number of independent sequences processed simultaneously. Larger batches provide more stable gradients but require more memory.
+
+### Block Size
+The maximum sequence length (context window) the model can process. Also called "context length". Default: 256 tokens.
+
+### BPE (Byte Pair Encoding)
+A tokenization algorithm that builds a vocabulary by iteratively merging frequent character pairs. GPT-2 uses BPE with ~50k tokens.
+
+## C
+
+### Causal Mask
+A triangular mask that prevents attention from seeing future tokens. Essential for autoregressive generation.
+
+### Checkpoint
+A saved snapshot of model weights and training state. Allows resuming training or using the model for inference.
+
+### Context Window
+See "Block Size". The number of tokens the model can "see" when making predictions.
+
+### Cross-Entropy Loss
+The training objective. Measures how well the model's predicted probability distribution matches the actual next token.
+
+## D
+
+### Decoder-Only
+A transformer architecture that only uses the decoder stack (with causal masking). GPT models are decoder-only, unlike encoder-decoder models like T5.
+
+### Dropout
+A regularization technique that randomly zeros out activations during training. Prevents overfitting. Set to 0 during inference.
+
+## E
+
+### Embedding
+A learned vector representation of a discrete token. Maps vocabulary indices to continuous vectors of dimension `n_embd`.
+
+### Embedding Dimension (n_embd)
+The size of token representations. Larger dimensions allow more expressive representations but require more computation. Default: 384.
+
+## F
+
+### Flash Attention
+An optimized attention implementation that's faster and more memory-efficient. Used when dropout is 0.
+
+### Forward Pass
+Computing the model's output from input. Propagates activations through all layers to produce logits.
+
+## G
+
+### GELU (Gaussian Error Linear Unit)
+An activation function: `GELU(x) = x * Φ(x)` where Φ is the Gaussian CDF. Smoother than ReLU.
+
+### Gradient Accumulation
+Simulating larger batch sizes by accumulating gradients over multiple forward-backward passes before updating weights.
+
+### Gradient Clipping
+Limiting gradient magnitudes to prevent training instability. Default max norm: 1.0.
+
+## H
+
+### Head (Attention Head)
+One parallel attention computation. Multi-head attention runs `n_head` heads in parallel, each with dimension `n_embd/n_head`.
+
+### Head Size
+The dimension of each attention head: `head_size = n_embd / n_head`.
+
+## I
+
+### Inference
+Using a trained model to generate predictions. Also called "sampling" or "generation".
+
+### Iteration
+One update step in training. May include multiple micro-batches if using gradient accumulation.
+
+## K
+
+### Key (K)
+In attention, the vector that represents "what information this position contains". Compared against queries.
+
+## L
+
+### Layer Normalization
+Normalizes activations across the feature dimension. Stabilizes training. Applied before attention and MLP in pre-norm architecture.
+
+### Learning Rate
+Controls how much weights change per update. Typically uses warmup + cosine decay schedule.
+
+### Logits
+Raw (unnormalized) model outputs before softmax. Shape: [batch, seq_len, vocab_size].
+
+### Loss
+The training objective to minimize. Cross-entropy between predicted and actual next tokens.
+
+## M
+
+### MFU (Model FLOPs Utilization)
+Percentage of theoretical GPU compute being used. Higher is better. Measures hardware efficiency.
+
+### MLP (Multi-Layer Perceptron)
+The feed-forward network in each transformer block. Expands to 4× dimension then projects back.
+
+### MPS (Metal Performance Shaders)
+Apple's GPU compute framework. Used for acceleration on Apple Silicon Macs.
+
+## N
+
+### n_embd
+Embedding dimension. The size of token vectors throughout the model. Default: 384.
+
+### n_head
+Number of attention heads. Must divide n_embd evenly. Default: 6.
+
+### n_layer
+Number of transformer blocks stacked. More layers = more capacity but slower. Default: 6.
+
+## O
+
+### Optimizer
+Algorithm that updates model weights based on gradients. nanoGPT uses AdamW.
+
+## P
+
+### Parameters
+The learnable weights of the model. Measured in millions (M) or billions (B).
+
+### Position Embedding
+Learned vectors that encode token position. Added to token embeddings so the model knows token order.
+
+### Pre-norm
+Architecture where LayerNorm is applied before (not after) attention/MLP. Improves training stability.
+
+## Q
+
+### Query (Q)
+In attention, the vector that represents "what this position is looking for". Compared against keys.
+
+## R
+
+### Residual Connection
+Adding the input directly to the output: `output = layer(x) + x`. Helps gradient flow and enables deeper networks.
+
+## S
+
+### Sampling
+Generating text by repeatedly predicting and appending tokens. See "Generation".
+
+### Softmax
+Converts logits to probabilities: `softmax(x)_i = exp(x_i) / sum(exp(x_j))`. Output sums to 1.
+
+### State Dict
+A dictionary mapping parameter names to their tensor values. Used for saving/loading models.
+
+## T
+
+### Temperature
+Scaling factor for logits before softmax during generation. Lower = more deterministic, higher = more random.
+
+### Token
+The atomic unit of text. Can be a character, word piece, or subword depending on the tokenizer.
+
+### Tokenizer
+Converts text to token IDs and vice versa. nanoGPT supports character-level and GPT-2 BPE tokenization.
+
+### Top-k Sampling
+Restricting sampling to the k most probable tokens. Reduces incoherent outputs.
+
+### Transformer
+The neural network architecture based on self-attention. Introduced in "Attention Is All You Need" (2017).
+
+## V
+
+### Value (V)
+In attention, the vector containing "the information to retrieve". Weighted sum of values is the attention output.
+
+### Validation Loss
+Loss computed on held-out data. Used to detect overfitting and decide when to save checkpoints.
+
+### Vocab Size
+Number of unique tokens. Character-level: ~65-100. GPT-2 BPE: 50,257.
+
+## W
+
+### Warmup
+Gradually increasing learning rate at the start of training. Prevents early instability.
+
+### Weight Decay
+L2 regularization applied to weights (not biases/layernorm). Prevents overfitting. Default: 0.1.
+
+### Weight Tying
+Sharing the token embedding matrix with the output projection. Reduces parameters.
+
+---
+
+## Model Size Formula
+
+```
+Parameters ≈ 12 × n_layer × n_embd²
+
+Example (default config):
+  12 × 6 × 384² ≈ 10.6M parameters
+```
+
+## Compute Requirements
+
+Training one iteration processes:
+```
+tokens = batch_size × block_size × gradient_accumulation_steps
+
+FLOPs ≈ 6 × parameters × tokens  (forward + backward)
+```
+
+## References
+
+- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) - Original transformer paper
+- [GPT-2](https://openai.com/research/better-language-models) - OpenAI's GPT-2 model
+- [nanoGPT](https://github.com/karpathy/nanoGPT) - Original Python implementation
+- [PaLM](https://arxiv.org/abs/2204.02311) - MFU calculation reference