nanogpt 0.1.2 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cf308fcec8ccec074200361b2327a2381a5412b92497391bb71ec5f154cd1283
-  data.tar.gz: 87dcc389df03af0ac59fc0e75bbef7be7071f00613a2662cecf365ba38bc853e
+  metadata.gz: f3563f92bd2b986ae27476c80c596cca77438d899b53d017f6cb708179fd7ad4
+  data.tar.gz: d15ed5effb810c0fa022b4fa3d6708fc2623332b0d3f4de5057a66343a9a3b2f
 SHA512:
-  metadata.gz: 2b6ceeb10236b639c82398c94d3c1a876eff549a17289ff45abae489e480a3d6a1db45f95a4b2e56f1d1df6afde28159965b188342d1e38c2482359dfb11e061
-  data.tar.gz: 0fd4653c2719d1d3c339a904437fd0657f17e49fa0a9d4361a3a259806fab8dc798ed7f179722b41913d3471e8799abf78823f207988987d86cba680d7f90f03
+  metadata.gz: 030bd007f053f6400afec46810e10d1c8a96b881df932c291fc7efb637a4c00424cd9304facac71e56783a861417dcd6ec247b2f6df3926a443d2540ecf6adcf
+  data.tar.gz: c0bcf4eae610c2475c841c186ee8a2a6aebc2bb22ecf2d0fbb9d65c7218152699598d320cc6972262ff918d0991c4cc4c951a68552bec46b24bd4fee10aa1b07

data/CHANGELOG.md

@@ -0,0 +1,45 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.2.0] - 2025-12-08
+
+### Added
+
+- **Custom text file training**: New `nanogpt prepare textfile <path>` command to train on any text file with character-level tokenization
+  - Streams through large files without loading everything into memory
+  - Auto-detects file encoding (UTF-8 or Windows-1252)
+  - Configurable output directory name (`--output=NAME`)
+  - Configurable train/validation split ratio (`--val_ratio=F`, default 0.1)
+- Updated README with documentation for training on custom text files
+
+## [0.1.2] - 2025-12-07
+
+### Fixed
+
+- Fixed `prepare` command to output files to current working directory
+
+## [0.1.1] - 2025-12-07
+
+### Fixed
+
+- Fixed typo in documentation
+
+## [0.1.0] - 2025-12-07
+
+### Added
+
+- Initial release
+- Full GPT-2 architecture implementation in Ruby
+- MPS (Metal) and CUDA GPU acceleration via torch.rb
+- Flash attention support when dropout=0
+- Character-level and GPT-2 BPE tokenizers
+- Cosine learning rate schedule with warmup
+- Gradient accumulation for larger effective batch sizes
+- Checkpointing and training resumption
+- Shakespeare character-level dataset
+- OpenWebText dataset support
+- CLI commands: `train`, `sample`, `bench`, `prepare`

data/Gemfile.lock

@@ -1,17 +1,35 @@
 PATH
   remote: .
   specs:
-    nanogpt (0.1.0)
+    nanogpt (0.3.0)
       numo-narray (~> 0.9)
+      rackup (~> 2.0)
+      sinatra (~> 4.0)
+      sqlite3 (~> 2.0)
       tiktoken_ruby (~> 0.0)
       torch-rb (~> 0.14)
+      webrick (~> 1.8)
 
 GEM
   remote: https://rubygems.org/
   specs:
+    base64 (0.3.0)
     diff-lcs (1.6.2)
+    logger (1.7.0)
+    mustermann (3.0.4)
+      ruby2_keywords (~> 0.0.1)
     numo-narray (0.9.2.1)
     parquet (0.7.3-arm64-darwin)
+    rack (3.2.5)
+    rack-protection (4.2.1)
+      base64 (>= 0.1.0)
+      logger (>= 1.6.0)
+      rack (>= 3.0.0, < 4)
+    rack-session (2.1.1)
+      base64 (>= 0.1.0)
+      rack (>= 3.0.0)
+    rackup (2.3.1)
+      rack (>= 3)
     rice (4.7.1)
     rspec (3.13.2)
       rspec-core (~> 3.13.0)
@@ -26,12 +44,24 @@ GEM
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.13.0)
     rspec-support (3.13.6)
+    ruby2_keywords (0.0.5)
+    sinatra (4.2.1)
+      logger (>= 1.6.0)
+      mustermann (~> 3.0)
+      rack (>= 3.0.0, < 4)
+      rack-protection (= 4.2.1)
+      rack-session (>= 2.0.0, < 3)
+      tilt (~> 2.0)
+    sqlite3 (2.9.0-arm64-darwin)
     tiktoken_ruby (0.0.13-arm64-darwin)
+    tilt (2.7.0)
     torch-rb (0.22.2)
       rice (>= 4.7)
+    webrick (1.9.2)
 
 PLATFORMS
   arm64-darwin-24
+  arm64-darwin-25
 
 DEPENDENCIES
   nanogpt!

data/README.md

@@ -1,5 +1,7 @@
 # nanoGPT
 
+[![Gem Version](https://badge.fury.io/rb/nanogpt.svg)](https://rubygems.org/gems/nanogpt)
+
 A Ruby port of Karpathy's [nanoGPT](https://github.com/karpathy/nanoGPT). Train GPT-2 style language models from scratch using [torch.rb](https://github.com/ankane/torch.rb).
 
 Built for Ruby developers who want to understand how LLMs work by building one.
@@ -13,7 +15,7 @@ gem install nanogpt
 nanogpt prepare shakespeare_char
 
 # Train (use MPS on Apple Silicon for 17x speedup)
-nanogpt train --dataset=shakespeare_char --device=mps
+nanogpt train --dataset=shakespeare_char --device=mps --max_iters=2000
 
 # Generate text
 nanogpt sample --dataset=shakespeare_char
@@ -30,7 +32,7 @@ bundle install
 bundle exec ruby data/shakespeare_char/prepare.rb
 
 # Train
-bundle exec exe/nanogpt train --dataset=shakespeare_char --device=mps
+bundle exec exe/nanogpt train --dataset=shakespeare_char --device=mps --max_iters=2000
 
 # Sample
 bundle exec exe/nanogpt sample --dataset=shakespeare_char
@@ -81,6 +83,53 @@ nanogpt bench [options]    # Run performance benchmarks
 --top_k=N            # Top-k sampling (default: 200)
 ```
 
+## Training on Your Own Text
+
+You can train on any text file using the `textfile` command:
+
+```bash
+# Prepare your text file (creates char-level tokenizer)
+nanogpt prepare textfile /path/to/mybook.txt --output=mybook
+
+# Train a model
+nanogpt train --dataset=mybook --device=mps --max_iters=2000
+
+# Generate text
+nanogpt sample --dataset=mybook --start="Once upon a time"
+```
+
+### Options
+
+```bash
+--output=NAME       # Output directory name (default: derived from filename)
+--val_ratio=F       # Validation split ratio (default: 0.1)
+```
+
+### Example: Training on a Novel
+
+```bash
+# Download a book
+curl -o lotr.txt "https://example.com/fellowship.txt"
+
+# Prepare (handles UTF-8 and Windows-1252 encodings)
+nanogpt prepare textfile lotr.txt --output=lotr
+
+# Train a larger model for better results
+nanogpt train --dataset=lotr --device=mps \
+  --max_iters=2000 \
+  --n_layer=6 --n_head=6 --n_embd=384 \
+  --block_size=256 --batch_size=32
+
+# Sample with a prompt
+nanogpt sample --dataset=lotr --start="Frodo" --max_new_tokens=500
+```
+
+The `textfile` command:
+- Streams through large files without loading everything into memory
+- Auto-detects encoding (UTF-8 or Windows-1252)
+- Creates a character-level vocabulary from your text
+- Splits into train/validation sets
+
 ## Features
 
 - Full GPT-2 architecture (attention, MLP, layer norm, embeddings)

data/docs/ARCHITECTURE.md

@@ -0,0 +1,429 @@
+# nanoGPT Architecture
+
+This document explains the architecture of nanoGPT, a Ruby implementation of GPT-2 style language models.
+
+## Overview
+
+nanoGPT implements a **decoder-only transformer** architecture for autoregressive language modeling. The model learns to predict the next token in a sequence given all previous tokens.
+
+```
+Input Tokens → Embeddings → Transformer Blocks → Output Logits → Next Token Prediction
+```
+
+## Core Components
+
+### 1. Token & Position Embeddings
+
+The model converts discrete tokens into continuous vectors:
+
+```
+Token IDs [0, 42, 7, ...] → Token Embeddings (vocab_size × n_embd)
+Position [0, 1, 2, ...]   → Position Embeddings (block_size × n_embd)
+
+Final Embedding = TokenEmbed(token) + PosEmbed(position)
+```
+
+**Files:** `lib/nano_gpt/model.rb:16-18`
+
+The position embeddings are learned (not sinusoidal) and allow the model to understand token order.
+
+### 2. Transformer Block
+
+Each transformer block contains two sub-layers with residual connections:
+
+```
+┌─────────────────────────────────────────────────┐
+│                Transformer Block                │
+├─────────────────────────────────────────────────┤
+│                                                 │
+│   x ─→ LayerNorm ─→ Attention ─→ (+) ─┐        │
+│   └────────────────────────────────────┘        │
+│                                                 │
+│   x ─→ LayerNorm ─→ MLP ─→ (+) ─→ output       │
+│   └─────────────────────────────────────┘       │
+│                                                 │
+└─────────────────────────────────────────────────┘
+```
+
+**File:** `lib/nano_gpt/layers/block.rb`
+
+This is "pre-norm" architecture (LayerNorm before sub-layer, not after) which improves training stability.
+
+### 3. Causal Self-Attention
+
+The attention mechanism allows each token to attend to all previous tokens (but not future ones):
+
+```
+Q = x @ W_q    (Query: what am I looking for?)
+K = x @ W_k    (Key: what do I contain?)
+V = x @ W_v    (Value: what information do I provide?)
+
+Attention(Q,K,V) = softmax(Q @ K^T / sqrt(d_k)) @ V
+```
+
+The "causal" part means we mask future positions:
+
+```
+Position:    0   1   2   3
+Token 0:    [1   0   0   0]  ← can only see itself
+Token 1:    [1   1   0   0]  ← can see positions 0,1
+Token 2:    [1   1   1   0]  ← can see positions 0,1,2
+Token 3:    [1   1   1   1]  ← can see all
+```
+
+**File:** `lib/nano_gpt/layers/causal_self_attention.rb`
+
+Multi-head attention splits the embedding into `n_head` parallel attention operations, allowing the model to attend to different aspects simultaneously.
+
+### 4. MLP (Feed-Forward Network)
+
+After attention, each token passes through an MLP:
+
+```
+x → Linear(n_embd → 4×n_embd) → GELU → Linear(4×n_embd → n_embd) → output
+```
+
+**File:** `lib/nano_gpt/layers/mlp.rb`
+
+The 4× expansion provides additional capacity for computation.
+
+### 5. Output Layer (Weight Tying)
+
+The final output uses the transposed token embedding matrix:
+
+```
+logits = LayerNorm(x) @ TokenEmbed.T
+```
+
+This "weight tying" reduces parameters and improves performance.
+
+## Data Flow
+
+```
+┌────────────────────────────────────────────────────────────────────┐
+│                        GPT Forward Pass                            │
+├────────────────────────────────────────────────────────────────────┤
+│                                                                    │
+│  Input: token_ids [batch, seq_len]                                 │
+│           ↓                                                        │
+│  Token Embedding + Position Embedding                              │
+│           ↓                                                        │
+│  Dropout                                                           │
+│           ↓                                                        │
+│  ┌──────────────────────────────────────┐                          │
+│  │ Transformer Block 1                  │                          │
+│  │   LayerNorm → Attention → Residual   │                          │
+│  │   LayerNorm → MLP → Residual         │                          │
+│  └──────────────────────────────────────┘                          │
+│           ↓                                                        │
+│  ... (repeat n_layer times) ...                                    │
+│           ↓                                                        │
+│  Final LayerNorm                                                   │
+│           ↓                                                        │
+│  Linear projection (tied weights)                                  │
+│           ↓                                                        │
+│  Output: logits [batch, seq_len, vocab_size]                       │
+│                                                                    │
+└────────────────────────────────────────────────────────────────────┘
+```
+
+## Training Pipeline
+
+```
+┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
+│   Prepare   │ →  │    Load     │ →  │   Forward   │ →  │  Backward   │
+│    Data     │    │   Batch     │    │    Pass     │    │    Pass     │
+└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
+      │                  │                  │                  │
+   train.bin        DataLoader           GPT.forward      loss.backward
+   val.bin          get_batch()           ↓                   ↓
+   meta.json           ↓               Cross-entropy      Gradients
+                   [x, y] tensors         loss             computed
+                                                              │
+                                                              ↓
+┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
+│   Update    │ ←  │    Clip     │ ←  │  Accumulate │ ←  │             │
+│   Weights   │    │  Gradients  │    │  Gradients  │    │             │
+└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
+      │
+   AdamW
+   optimizer
+```
+
+**File:** `lib/nano_gpt/trainer.rb`
+
+## Generation (Inference)
+
+```
+Start: "Hello"
+        ↓
+Tokenize: [15496]
+        ↓
+┌─────────────────────────────────────┐
+│ Loop until max_tokens:              │
+│   1. Forward pass → logits          │
+│   2. Apply temperature              │
+│   3. Apply top-k filtering          │
+│   4. Sample from distribution       │
+│   5. Append to sequence             │
+└─────────────────────────────────────┘
+        ↓
+Decode: "Hello, how are you today?"
+```
+
+**File:** `lib/nano_gpt/model.rb:110-147`
+
+## File Structure
+
+```
+lib/nano_gpt/
+├── model.rb              # GPT model class
+├── config.rb             # GPTConfig (model hyperparameters)
+├── train_config.rb       # Training/sampling/bench configs
+├── trainer.rb            # Training loop
+├── data_loader.rb        # Batch loading from binary files
+├── tokenizer.rb          # Character & GPT-2 BPE tokenizers
+├── lr_scheduler.rb       # Cosine annealing with warmup
+├── device.rb             # CPU/CUDA/MPS device detection
+├── textfile_preparer.rb  # Custom dataset preparation
+└── layers/
+    ├── block.rb                  # Transformer block
+    ├── causal_self_attention.rb  # Multi-head attention
+    ├── mlp.rb                    # Feed-forward network
+    └── layer_norm.rb             # Layer normalization
+```
+
+---
+
+# Glossary
+
+## A
+
+### Attention
+A mechanism that allows each position in a sequence to dynamically focus on other positions. Computes weighted sums of values based on query-key similarities.
+
+### Autoregressive
+A model that generates outputs one step at a time, where each step depends on all previous steps. GPT generates text left-to-right, predicting one token at a time.
+
+### AdamW
+An optimizer that combines Adam (adaptive learning rates) with decoupled weight decay. Used for training the model.
+
+## B
+
+### Batch Size
+The number of independent sequences processed simultaneously. Larger batches provide more stable gradients but require more memory.
+
+### Block Size
+The maximum sequence length (context window) the model can process. Also called "context length". Default: 256 tokens.
+
+### BPE (Byte Pair Encoding)
+A tokenization algorithm that builds a vocabulary by iteratively merging frequent character pairs. GPT-2 uses BPE with ~50k tokens.
+
+## C
+
+### Causal Mask
+A triangular mask that prevents attention from seeing future tokens. Essential for autoregressive generation.
+
+### Checkpoint
+A saved snapshot of model weights and training state. Allows resuming training or using the model for inference.
+
+### Context Window
+See "Block Size". The number of tokens the model can "see" when making predictions.
+
+### Cross-Entropy Loss
+The training objective. Measures how well the model's predicted probability distribution matches the actual next token.
+
+## D
+
+### Decoder-Only
+A transformer architecture that only uses the decoder stack (with causal masking). GPT models are decoder-only, unlike encoder-decoder models like T5.
+
+### Dropout
+A regularization technique that randomly zeros out activations during training. Prevents overfitting. Set to 0 during inference.
+
+## E
+
+### Embedding
+A learned vector representation of a discrete token. Maps vocabulary indices to continuous vectors of dimension `n_embd`.
+
+### Embedding Dimension (n_embd)
+The size of token representations. Larger dimensions allow more expressive representations but require more computation. Default: 384.
+
+## F
+
+### Flash Attention
+An optimized attention implementation that's faster and more memory-efficient. Used when dropout is 0.
+
+### Forward Pass
+Computing the model's output from input. Propagates activations through all layers to produce logits.
+
+## G
+
+### GELU (Gaussian Error Linear Unit)
+An activation function: `GELU(x) = x * Φ(x)` where Φ is the Gaussian CDF. Smoother than ReLU.
+
+### Gradient Accumulation
+Simulating larger batch sizes by accumulating gradients over multiple forward-backward passes before updating weights.
+
+### Gradient Clipping
+Limiting gradient magnitudes to prevent training instability. Default max norm: 1.0.
+
+## H
+
+### Head (Attention Head)
+One parallel attention computation. Multi-head attention runs `n_head` heads in parallel, each with dimension `n_embd/n_head`.
+
+### Head Size
+The dimension of each attention head: `head_size = n_embd / n_head`.
+
+## I
+
+### Inference
+Using a trained model to generate predictions. Also called "sampling" or "generation".
+
+### Iteration
+One update step in training. May include multiple micro-batches if using gradient accumulation.
+
+## K
+
+### Key (K)
+In attention, the vector that represents "what information this position contains". Compared against queries.
+
+## L
+
+### Layer Normalization
+Normalizes activations across the feature dimension. Stabilizes training. Applied before attention and MLP in pre-norm architecture.
+
+### Learning Rate
+Controls how much weights change per update. Typically uses warmup + cosine decay schedule.
+
+### Logits
+Raw (unnormalized) model outputs before softmax. Shape: [batch, seq_len, vocab_size].
+
+### Loss
+The training objective to minimize. Cross-entropy between predicted and actual next tokens.
+
+## M
+
+### MFU (Model FLOPs Utilization)
+Percentage of theoretical GPU compute being used. Higher is better. Measures hardware efficiency.
+
+### MLP (Multi-Layer Perceptron)
+The feed-forward network in each transformer block. Expands to 4× dimension then projects back.
+
+### MPS (Metal Performance Shaders)
+Apple's GPU compute framework. Used for acceleration on Apple Silicon Macs.
+
+## N
+
+### n_embd
+Embedding dimension. The size of token vectors throughout the model. Default: 384.
+
+### n_head
+Number of attention heads. Must divide n_embd evenly. Default: 6.
+
+### n_layer
+Number of transformer blocks stacked. More layers = more capacity but slower. Default: 6.
+
+## O
+
+### Optimizer
+Algorithm that updates model weights based on gradients. nanoGPT uses AdamW.
+
+## P
+
+### Parameters
+The learnable weights of the model. Measured in millions (M) or billions (B).
+
+### Position Embedding
+Learned vectors that encode token position. Added to token embeddings so the model knows token order.
+
+### Pre-norm
+Architecture where LayerNorm is applied before (not after) attention/MLP. Improves training stability.
+
+## Q
+
+### Query (Q)
+In attention, the vector that represents "what this position is looking for". Compared against keys.
+
+## R
+
+### Residual Connection
+Adding the input directly to the output: `output = layer(x) + x`. Helps gradient flow and enables deeper networks.
+
+## S
+
+### Sampling
+Generating text by repeatedly predicting and appending tokens. See "Generation".
+
+### Softmax
+Converts logits to probabilities: `softmax(x)_i = exp(x_i) / sum(exp(x_j))`. Output sums to 1.
+
+### State Dict
+A dictionary mapping parameter names to their tensor values. Used for saving/loading models.
+
+## T
+
+### Temperature
+Scaling factor for logits before softmax during generation. Lower = more deterministic, higher = more random.
+
+### Token
+The atomic unit of text. Can be a character, word piece, or subword depending on the tokenizer.
+
+### Tokenizer
+Converts text to token IDs and vice versa. nanoGPT supports character-level and GPT-2 BPE tokenization.
+
+### Top-k Sampling
+Restricting sampling to the k most probable tokens. Reduces incoherent outputs.
+
+### Transformer
+The neural network architecture based on self-attention. Introduced in "Attention Is All You Need" (2017).
+
+## V
+
+### Value (V)
+In attention, the vector containing "the information to retrieve". Weighted sum of values is the attention output.
+
+### Validation Loss
+Loss computed on held-out data. Used to detect overfitting and decide when to save checkpoints.
+
+### Vocab Size
+Number of unique tokens. Character-level: ~65-100. GPT-2 BPE: 50,257.
+
+## W
+
+### Warmup
+Gradually increasing learning rate at the start of training. Prevents early instability.
+
+### Weight Decay
+L2 regularization applied to weights (not biases/layernorm). Prevents overfitting. Default: 0.1.
+
+### Weight Tying
+Sharing the token embedding matrix with the output projection. Reduces parameters.
+
+---
+
+## Model Size Formula
+
+```
+Parameters ≈ 12 × n_layer × n_embd²
+
+Example (default config):
+  12 × 6 × 384² ≈ 10.6M parameters
+```
+
+## Compute Requirements
+
+Training one iteration processes:
+```
+tokens = batch_size × block_size × gradient_accumulation_steps
+
+FLOPs ≈ 6 × parameters × tokens  (forward + backward)
+```
+
+## References
+
+- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) - Original transformer paper
+- [GPT-2](https://openai.com/research/better-language-models) - OpenAI's GPT-2 model
+- [nanoGPT](https://github.com/karpathy/nanoGPT) - Original Python implementation
+- [PaLM](https://arxiv.org/abs/2204.02311) - MFU calculation reference