oglodon-transformer 0.4.0__tar.gz

oglodon_transformer-0.4.0/.gitignore

@@ -0,0 +1,21 @@
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+.venv/
+*.pt
+checkpoint*/
+site/
+.mypy_cache/
+*.pyc
+.DS_Store
+Thumbs.db
+.vscode/
+.idea/
+.ipynb_checkpoints/
+oglodon_runs/
+runs/
+*.bin
+*.safetensors

oglodon_transformer-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Oglodon contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

oglodon_transformer-0.4.0/PKG-INFO

@@ -0,0 +1,168 @@
+Metadata-Version: 2.5
+Name: oglodon-transformer
+Version: 0.4.0
+Summary: Oglodon: build, train and behaviour-verify Transformers; `oglodon verify` checks any causal LM (Hugging Face, custom PyTorch) for causality, padding, KV-cache, numerics and gradient bugs.
+Project-URL: Homepage, https://github.com/Dhruvg0/Oglodon
+Project-URL: Issues, https://github.com/Dhruvg0/Oglodon/issues
+Author-email: Dhruv Puri <dhruvpurigoswami2004@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: education,gpt,gqa,kv-cache,llm,pytorch,rope,transformer
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: numpy>=1.26
+Requires-Dist: safetensors>=0.4
+Requires-Dist: torch>=2.2
+Provides-Extra: dev
+Requires-Dist: build>=1.0; extra == 'dev'
+Requires-Dist: matplotlib>=3.8; extra == 'dev'
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: pyyaml>=6; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: transformers>=4.46; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.6; extra == 'docs'
+Provides-Extra: hf
+Requires-Dist: transformers>=4.46; extra == 'hf'
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.8; extra == 'viz'
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6; extra == 'yaml'
+Description-Content-Type: text/markdown
+
+# Oglodon
+
+**Build. Train. Understand. Verify.**
+
+[![Python](https://img.shields.io/badge/python-3.11%2B-3776AB)](https://www.python.org/)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.2%2B-EE4C2C)](https://pytorch.org/)
+[![License](https://img.shields.io/badge/license-MIT-yellow)](https://github.com/Dhruvg0/oglodon/blob/main/LICENSE)
+[![Tests](https://github.com/Dhruvg0/oglodon/actions/workflows/tests.yml/badge.svg)](https://github.com/Dhruvg0/oglodon/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/oglodon-transformer)](https://pypi.org/project/oglodon-transformer/)
+
+Oglodon is a small PyTorch library for building, training and checking decoder-only Transformers (GPT-style, with
+RoPE, RMSNorm, SwiGLU, grouped-query attention and a KV cache).
+
+Its most useful part is **`oglodon verify`**: it tests *any* causal language model (Hugging Face, your own PyTorch
+module, or Oglodon) for bugs that still train but quietly make the model worse, such as a token seeing the future,
+padding changing the answer, or a KV cache that disagrees with a full forward pass.
+
+> **Status: alpha, for learning and research at small scale.** Not production-ready and not a replacement for large-scale
+> frameworks. See [what has and has not been tested](#what-has-been-tested).
+
+## Install
+
+```bash
+pip install oglodon-transformer
+```
+
+Needs Python 3.11+ and PyTorch. Optional extras: `[hf]` Hugging Face import, export and verification, `[viz]` plots,
+`[yaml]` YAML configs, `[dev]` tests and linters. Working from a clone: `pip install -e ".[dev]"`.
+
+## Try it in two minutes
+
+Put a few `.txt` or `.md` files in a folder, then:
+
+```bash
+oglodon train my_texts/ --out run1 --steps 500
+oglodon generate run1 --prompt "Once upon a time" --seed 1
+oglodon eval run1 --text my_texts/some_file.txt
+```
+
+`train` builds a tokenizer, trains a small model and saves a checkpoint; `eval` prints perplexity next to the score of
+a model that guesses at random, so you can see whether it learned anything. A model trained for minutes learns your
+text's vocabulary and local patterns, not fluent language. No Python code is needed.
+
+## How it fits together
+
+```text
+ text files ──► BPE tokenizer ──► token file ──► train ──► CHECKPOINT
+                                                              │
+        ┌─────────────┬──────────────┬──────────────┬─────────┴────────┐
+        ▼             ▼              ▼              ▼                  ▼
+    generate         eval          verify        quantize         export to
+  (stream text)  (perplexity)   (behaviour)    (int8 copy)      Hugging Face
+                                     ▲
+ any Hugging Face / PyTorch model ───┘     verify accepts any causal LM
+```
+
+Inside the model:
+
+```text
+tokens ─► Embedding ─► [ Norm ─► Attention (RoPE, KV cache) ─► + ─► Norm ─► FFN ─► + ] × N ─► Norm ─► LM head ─► logits
+                                MHA / GQA / MQA                          SwiGLU / GELU
+```
+
+## What you can do
+
+| Command | What it does |
+|---|---|
+| `oglodon train` | Text files to tokenizer, trained model and report. Exact `--resume`, data-parallel under `torchrun` |
+| `oglodon generate` | Streams text: temperature, top-k, top-p, min-p, repetition penalty, stop strings, KV cache |
+| `oglodon eval` | Perplexity, multiple-choice accuracy, generation sanity checks, saved as JSON |
+| `oglodon verify` | Behaviour checks on any causal LM; `--cross-check` compares against a Hugging Face model |
+| `oglodon validate` | Compares an Oglodon model with an independent float64 reference implementation |
+| `oglodon compare` | Trains variants on identical data and seeds and writes reproducible comparison reports |
+| `oglodon quantize` | Int8 weight-only copy of a checkpoint with a measured accuracy drift |
+| `oglodon info` | Versions and whether CUDA is available |
+
+Architecture options: MHA, GQA or MQA attention, RoPE, learned or ALiBi positions, RMSNorm or LayerNorm, SwiGLU,
+GeGLU, GELU or ReLU² feed-forward, sliding window, qkv bias. Export to Llama, Mistral or Qwen2 format for
+Hugging Face with `export_hf(model, "hf_dir")`.
+
+## Verify any model
+
+```bash
+oglodon verify --checkpoint run1                      # an Oglodon checkpoint
+oglodon verify --hf HuggingFaceTB/SmolLM2-135M --cross-check --dtype float32
+```
+
+| Check | What it catches |
+|---|---|
+| Causality | A token's prediction changes when later tokens change |
+| Padding and batching | Padded or batched inputs giving different answers than single ones |
+| KV cache | Cached decoding disagreeing with a full forward pass |
+| Numerics | float32 vs float64 drift, non-finite values |
+| Gradients | Autograd disagreeing with finite differences, dead parameters |
+| Cross-check | An Oglodon model and its Hugging Face twin producing different logits |
+
+Skipped or unsupported checks are reported as such and are never counted as passed.
+
+## What has been tested
+
+| Verified | Not yet |
+|---|---|
+| Test suite on CPU: Python 3.11, 3.12 and 3.13 (Linux) and the suite on Windows | macOS |
+| Test suite, `validate`, `verify`, bf16 and fp16 steps on one Tesla T4 (Colab) | Multi-GPU, FlashAttention kernels, other GPUs |
+| `verify --cross-check` on SmolLM2-135M in float32: logits agree to about 1e-4, greedy output identical | Other pretrained models |
+| Int8 drift measured on a small trained model (perplexity change about 0.001%) | Int8 on large models or on a GPU |
+| Hugging Face export matches logits to about 1e-6 | Gemma, Phi, Mixtral converters, FSDP, 4-bit quantization |
+
+Details and exact numbers: [Limitations](https://github.com/Dhruvg0/oglodon/blob/main/docs/limitations.md) and [Performance and scaling](https://github.com/Dhruvg0/oglodon/blob/main/docs/performance-and-scaling.md).
+
+## Documentation
+
+* [Getting started](https://github.com/Dhruvg0/oglodon/blob/main/docs/getting-started.md) and [command reference](https://github.com/Dhruvg0/oglodon/blob/main/docs/cli.md)
+* [Training](https://github.com/Dhruvg0/oglodon/blob/main/docs/training.md), [data and tokenizers](https://github.com/Dhruvg0/oglodon/blob/main/docs/data-and-tokenization.md), [generation](https://github.com/Dhruvg0/oglodon/blob/main/docs/generation.md),
+  [evaluation](https://github.com/Dhruvg0/oglodon/blob/main/docs/evaluation.md)
+* [Verify](https://github.com/Dhruvg0/oglodon/blob/main/docs/verify.md), [validation](https://github.com/Dhruvg0/oglodon/blob/main/docs/validation.md), [experiments](https://github.com/Dhruvg0/oglodon/blob/main/docs/experiments.md),
+  [quantization](https://github.com/Dhruvg0/oglodon/blob/main/docs/quantization.md), [Hugging Face interop](https://github.com/Dhruvg0/oglodon/blob/main/docs/interoperability.md)
+* [Python and feature reference](https://github.com/Dhruvg0/oglodon/blob/main/docs/reference.md), [architecture](https://github.com/Dhruvg0/oglodon/blob/main/docs/architecture.md),
+  [configuration](https://github.com/Dhruvg0/oglodon/blob/main/docs/configuration.md), [troubleshooting](https://github.com/Dhruvg0/oglodon/blob/main/docs/troubleshooting.md)
+* Runnable scripts in [`examples/`](https://github.com/Dhruvg0/oglodon/tree/main/examples)
+
+## Contributing and licence
+
+Contributions are welcome; see [docs/contributing.md](https://github.com/Dhruvg0/oglodon/blob/main/docs/contributing.md). Released under the [MIT licence](https://github.com/Dhruvg0/oglodon/blob/main/LICENSE).
+
+Oglodon is an independent project. It is not affiliated with, endorsed by or derived from NVIDIA or any hardware vendor.

oglodon_transformer-0.4.0/README.md

@@ -0,0 +1,126 @@
+# Oglodon
+
+**Build. Train. Understand. Verify.**
+
+[![Python](https://img.shields.io/badge/python-3.11%2B-3776AB)](https://www.python.org/)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.2%2B-EE4C2C)](https://pytorch.org/)
+[![License](https://img.shields.io/badge/license-MIT-yellow)](https://github.com/Dhruvg0/oglodon/blob/main/LICENSE)
+[![Tests](https://github.com/Dhruvg0/oglodon/actions/workflows/tests.yml/badge.svg)](https://github.com/Dhruvg0/oglodon/actions/workflows/tests.yml)
+[![PyPI](https://img.shields.io/pypi/v/oglodon-transformer)](https://pypi.org/project/oglodon-transformer/)
+
+Oglodon is a small PyTorch library for building, training and checking decoder-only Transformers (GPT-style, with
+RoPE, RMSNorm, SwiGLU, grouped-query attention and a KV cache).
+
+Its most useful part is **`oglodon verify`**: it tests *any* causal language model (Hugging Face, your own PyTorch
+module, or Oglodon) for bugs that still train but quietly make the model worse, such as a token seeing the future,
+padding changing the answer, or a KV cache that disagrees with a full forward pass.
+
+> **Status: alpha, for learning and research at small scale.** Not production-ready and not a replacement for large-scale
+> frameworks. See [what has and has not been tested](#what-has-been-tested).
+
+## Install
+
+```bash
+pip install oglodon-transformer
+```
+
+Needs Python 3.11+ and PyTorch. Optional extras: `[hf]` Hugging Face import, export and verification, `[viz]` plots,
+`[yaml]` YAML configs, `[dev]` tests and linters. Working from a clone: `pip install -e ".[dev]"`.
+
+## Try it in two minutes
+
+Put a few `.txt` or `.md` files in a folder, then:
+
+```bash
+oglodon train my_texts/ --out run1 --steps 500
+oglodon generate run1 --prompt "Once upon a time" --seed 1
+oglodon eval run1 --text my_texts/some_file.txt
+```
+
+`train` builds a tokenizer, trains a small model and saves a checkpoint; `eval` prints perplexity next to the score of
+a model that guesses at random, so you can see whether it learned anything. A model trained for minutes learns your
+text's vocabulary and local patterns, not fluent language. No Python code is needed.
+
+## How it fits together
+
+```text
+ text files ──► BPE tokenizer ──► token file ──► train ──► CHECKPOINT
+                                                              │
+        ┌─────────────┬──────────────┬──────────────┬─────────┴────────┐
+        ▼             ▼              ▼              ▼                  ▼
+    generate         eval          verify        quantize         export to
+  (stream text)  (perplexity)   (behaviour)    (int8 copy)      Hugging Face
+                                     ▲
+ any Hugging Face / PyTorch model ───┘     verify accepts any causal LM
+```
+
+Inside the model:
+
+```text
+tokens ─► Embedding ─► [ Norm ─► Attention (RoPE, KV cache) ─► + ─► Norm ─► FFN ─► + ] × N ─► Norm ─► LM head ─► logits
+                                MHA / GQA / MQA                          SwiGLU / GELU
+```
+
+## What you can do
+
+| Command | What it does |
+|---|---|
+| `oglodon train` | Text files to tokenizer, trained model and report. Exact `--resume`, data-parallel under `torchrun` |
+| `oglodon generate` | Streams text: temperature, top-k, top-p, min-p, repetition penalty, stop strings, KV cache |
+| `oglodon eval` | Perplexity, multiple-choice accuracy, generation sanity checks, saved as JSON |
+| `oglodon verify` | Behaviour checks on any causal LM; `--cross-check` compares against a Hugging Face model |
+| `oglodon validate` | Compares an Oglodon model with an independent float64 reference implementation |
+| `oglodon compare` | Trains variants on identical data and seeds and writes reproducible comparison reports |
+| `oglodon quantize` | Int8 weight-only copy of a checkpoint with a measured accuracy drift |
+| `oglodon info` | Versions and whether CUDA is available |
+
+Architecture options: MHA, GQA or MQA attention, RoPE, learned or ALiBi positions, RMSNorm or LayerNorm, SwiGLU,
+GeGLU, GELU or ReLU² feed-forward, sliding window, qkv bias. Export to Llama, Mistral or Qwen2 format for
+Hugging Face with `export_hf(model, "hf_dir")`.
+
+## Verify any model
+
+```bash
+oglodon verify --checkpoint run1                      # an Oglodon checkpoint
+oglodon verify --hf HuggingFaceTB/SmolLM2-135M --cross-check --dtype float32
+```
+
+| Check | What it catches |
+|---|---|
+| Causality | A token's prediction changes when later tokens change |
+| Padding and batching | Padded or batched inputs giving different answers than single ones |
+| KV cache | Cached decoding disagreeing with a full forward pass |
+| Numerics | float32 vs float64 drift, non-finite values |
+| Gradients | Autograd disagreeing with finite differences, dead parameters |
+| Cross-check | An Oglodon model and its Hugging Face twin producing different logits |
+
+Skipped or unsupported checks are reported as such and are never counted as passed.
+
+## What has been tested
+
+| Verified | Not yet |
+|---|---|
+| Test suite on CPU: Python 3.11, 3.12 and 3.13 (Linux) and the suite on Windows | macOS |
+| Test suite, `validate`, `verify`, bf16 and fp16 steps on one Tesla T4 (Colab) | Multi-GPU, FlashAttention kernels, other GPUs |
+| `verify --cross-check` on SmolLM2-135M in float32: logits agree to about 1e-4, greedy output identical | Other pretrained models |
+| Int8 drift measured on a small trained model (perplexity change about 0.001%) | Int8 on large models or on a GPU |
+| Hugging Face export matches logits to about 1e-6 | Gemma, Phi, Mixtral converters, FSDP, 4-bit quantization |
+
+Details and exact numbers: [Limitations](https://github.com/Dhruvg0/oglodon/blob/main/docs/limitations.md) and [Performance and scaling](https://github.com/Dhruvg0/oglodon/blob/main/docs/performance-and-scaling.md).
+
+## Documentation
+
+* [Getting started](https://github.com/Dhruvg0/oglodon/blob/main/docs/getting-started.md) and [command reference](https://github.com/Dhruvg0/oglodon/blob/main/docs/cli.md)
+* [Training](https://github.com/Dhruvg0/oglodon/blob/main/docs/training.md), [data and tokenizers](https://github.com/Dhruvg0/oglodon/blob/main/docs/data-and-tokenization.md), [generation](https://github.com/Dhruvg0/oglodon/blob/main/docs/generation.md),
+  [evaluation](https://github.com/Dhruvg0/oglodon/blob/main/docs/evaluation.md)
+* [Verify](https://github.com/Dhruvg0/oglodon/blob/main/docs/verify.md), [validation](https://github.com/Dhruvg0/oglodon/blob/main/docs/validation.md), [experiments](https://github.com/Dhruvg0/oglodon/blob/main/docs/experiments.md),
+  [quantization](https://github.com/Dhruvg0/oglodon/blob/main/docs/quantization.md), [Hugging Face interop](https://github.com/Dhruvg0/oglodon/blob/main/docs/interoperability.md)
+* [Python and feature reference](https://github.com/Dhruvg0/oglodon/blob/main/docs/reference.md), [architecture](https://github.com/Dhruvg0/oglodon/blob/main/docs/architecture.md),
+  [configuration](https://github.com/Dhruvg0/oglodon/blob/main/docs/configuration.md), [troubleshooting](https://github.com/Dhruvg0/oglodon/blob/main/docs/troubleshooting.md)
+* Runnable scripts in [`examples/`](https://github.com/Dhruvg0/oglodon/tree/main/examples)
+
+## Contributing and licence
+
+Contributions are welcome; see [docs/contributing.md](https://github.com/Dhruvg0/oglodon/blob/main/docs/contributing.md). Released under the [MIT licence](https://github.com/Dhruvg0/oglodon/blob/main/LICENSE).
+
+Oglodon is an independent project. It is not affiliated with, endorsed by or derived from NVIDIA or any hardware vendor.

oglodon_transformer-0.4.0/benchmarks/run_comparison.py

@@ -0,0 +1,35 @@
+"""Measure MHA vs GQA vs MQA on this machine (nothing here is a pre-recorded number).
+
+python benchmarks/run_comparison.py --hidden 256 --layers 6 --heads 8 --seq 128 --batch 4
+"""
+
+import argparse
+
+from oglodon import OglodonConfig
+from oglodon.benchmarks import compare_architectures, format_results
+
+p = argparse.ArgumentParser()
+p.add_argument("--hidden", type=int, default=256)
+p.add_argument("--layers", type=int, default=6)
+p.add_argument("--heads", type=int, default=8)
+p.add_argument("--seq", type=int, default=128)
+p.add_argument("--new-tokens", type=int, default=32)
+p.add_argument("--batch", type=int, default=4)
+p.add_argument("--repeats", type=int, default=5)
+a = p.parse_args()
+
+kv_options = {"mha": a.heads, "gqa": max(1, a.heads // 4), "mqa": 1}
+configs = [
+    OglodonConfig(
+        vocab_size=1000,
+        hidden_size=a.hidden,
+        num_layers=a.layers,
+        num_heads=a.heads,
+        num_kv_heads=kv,
+        max_seq_len=a.seq + a.new_tokens,
+    )
+    for kv in kv_options.values()
+]
+results = compare_architectures(configs, a.batch, a.seq, a.new_tokens, list(kv_options), a.repeats)
+print(format_results(results))
+print("\nforward/generate times and cache sizes are measured; 'kv (est)' is the analytic formula.")

oglodon_transformer-0.4.0/configs/compare_mha_gqa.yaml

@@ -0,0 +1,23 @@
+# oglodon compare configs/compare_mha_gqa.yaml --out comparison.json
+model:
+  vocab_size: 1            # replaced by the tokenizer's size
+  hidden_size: 64
+  num_layers: 2
+  num_heads: 8
+  num_kv_heads: 8
+  max_seq_len: 96
+variants:
+  mha: {num_kv_heads: 8}
+  gqa: {num_kv_heads: 2}
+  mqa: {num_kv_heads: 1}
+baseline: mha
+data:
+  block_size: 32           # built-in synthetic corpus; set `path: my.txt` for your own text
+  val_fraction: 0.1
+train:
+  steps: 300
+  batch_size: 16
+  learning_rate: 0.003
+  seeds: [0, 1, 2]
+  match_params: true
+  validate: true

oglodon_transformer-0.4.0/configs/compare_positions_ffn.yaml

@@ -0,0 +1,18 @@
+# Position methods and feed-forward variants on the same data and budget:
+#   oglodon compare configs/compare_positions_ffn.yaml --out positions.json --plot
+model:
+  vocab_size: 1
+  hidden_size: 64
+  num_layers: 2
+  num_heads: 8
+  num_kv_heads: 8
+  max_seq_len: 96
+variants:
+  rope: {}
+  learned: {use_rope: false}
+  alibi: {use_rope: false, use_alibi: true}
+  geglu: {ffn_activation: gelu}
+  relu2: {ffn_activation: relu2}
+baseline: rope
+data: {block_size: 32, val_fraction: 0.1}
+train: {steps: 200, batch_size: 16, learning_rate: 0.003, seeds: [0, 1, 2], match_params: true, validate: true}

oglodon_transformer-0.4.0/configs/gqa.yaml

@@ -0,0 +1,10 @@
+# oglodon compare configs/mha.yaml configs/gqa.yaml --out comparison.json
+model:
+  vocab_size: 1            # replaced by the tokenizer's size
+  hidden_size: 64
+  num_layers: 2
+  num_heads: 8
+  num_kv_heads: 2
+  max_seq_len: 96
+data: {block_size: 32, val_fraction: 0.1}
+train: {steps: 200, batch_size: 16, learning_rate: 0.003, seeds: [0, 1, 2], match_params: true, validate: true}

oglodon_transformer-0.4.0/configs/mha.yaml

@@ -0,0 +1,10 @@
+# oglodon compare configs/mha.yaml configs/gqa.yaml --out comparison.json
+model:
+  vocab_size: 1            # replaced by the tokenizer's size
+  hidden_size: 64
+  num_layers: 2
+  num_heads: 8
+  num_kv_heads: 8
+  max_seq_len: 96
+data: {block_size: 32, val_fraction: 0.1}
+train: {steps: 200, batch_size: 16, learning_rate: 0.003, seeds: [0, 1, 2], match_params: true, validate: true}

oglodon_transformer-0.4.0/configs/tiny_gqa.yaml

@@ -0,0 +1,14 @@
+# oglodon validate configs/tiny_gqa.yaml --out validation_report.json
+model:
+  vocab_size: 64
+  hidden_size: 64
+  num_layers: 2
+  num_heads: 8
+  num_kv_heads: 2        # GQA; 8 = MHA, 1 = MQA
+  max_seq_len: 64
+  eos_token_id: 1
+  pad_token_id: 0
+validation:
+  device: cpu
+  dtype: float32
+  seed: 0

oglodon_transformer-0.4.0/docs/architecture.md

@@ -0,0 +1,88 @@
+# Architecture
+
+```
+tokens (B,T)
+   |  embed_tokens                      (+ learned positions if use_rope=False and not use_alibi)
+   v
+x (B,T,d) --+--------------------------------------------------------------+
+            |   TransformerBlock x num_layers                              |
+            |     h = x + Attn(Norm1(x))      q,k RoPE (or ALiBi bias)     |
+            |     y = h + FFN(Norm2(h))       FFN = gated or plain MLP     |
+            +--------------------------------------------------------------+
+   |  final_norm
+   v
+lm_head (tied with embed_tokens by default)  ->  logits (B,T,V)
+```
+
+## Attention layouts
+
+`num_heads` query heads share `num_kv_heads` key/value heads; query head `h` reads K/V head
+`h // (num_heads // num_kv_heads)`.
+
+| layout | `num_kv_heads` | KV-cache size vs MHA |
+|---|---|---|
+| MHA | `num_heads` | 1x |
+| GQA | `1 < k < num_heads` | `k / num_heads` |
+| MQA | `1` | `1 / num_heads` |
+
+## Position methods and feed-forward variants
+
+| option | effect |
+|---|---|
+| `use_rope=True` (default) | rotary positions applied to q and k |
+| `use_rope=False` | learned absolute position embeddings |
+| `use_rope=False, use_alibi=True` | no position parameters; a linear distance penalty (per-head slope) is added to the attention scores |
+| `use_swiglu=True` with `ffn_activation` `silu` / `gelu` / `relu2` | gated feed-forward: SwiGLU / GeGLU / ReLU²-gated |
+| `use_swiglu=False` with `ffn_activation` `gelu` / `silu` / `relu2` | plain two-matrix MLP |
+
+With `attn_impl="sdpa"`, plain causal attention (no padding mask, window or ALiBi) calls PyTorch's fused attention with
+`is_causal=True`; every other case passes an explicit mask. Both give the same result (tested).
+
+## Package map
+
+| package | role |
+|---|---|
+| `oglodon.layers`, `oglodon.models` | attention, RoPE, norms, feed-forward, KV cache, the model |
+| `oglodon.generation` | sampling, streaming, stop criteria |
+| `oglodon.training` | `Trainer`, checkpoints, distributed helpers |
+| `oglodon.workflow` | text files to checkpoint (`train_from_files`), `complete` |
+| `oglodon.data`, `oglodon.tokenization` | token files, datasets, BPE/char tokenizers |
+| `oglodon.evaluation` | perplexity, multiple choice, generation checks |
+| `oglodon.validation` | float64 reference checks for Oglodon models |
+| `oglodon.verify` | model-agnostic behavioural checks, cross-implementation equivalence |
+| `oglodon.convert` | Hugging Face import and export |
+| `oglodon.experiments` | controlled comparisons and reports |
+
+## Where each notebook component went
+
+| MiniGPT notebook | Oglodon |
+|---|---|
+| `RMSNorm` | `oglodon.layers.normalization.RMSNorm` (+ `nn.LayerNorm` option) |
+| `rope_tables`, `apply_rope` | `oglodon.layers.rope.RotaryEmbedding`, `apply_rope` (same interleaved pairing) |
+| `Attention` (GQA, concat cache) | `oglodon.layers.attention.CausalSelfAttention` + `oglodon.layers.cache.KVCache` |
+| `SwiGLU` | `oglodon.layers.feedforward.SwiGLU` (+ `GELUMLP`) |
+| `Block`, `MiniGPT` | `oglodon.models.transformer.TransformerBlock`, `oglodon.models.gpt.Oglodon` |
+| char vocab, `enc`/`dec` | `oglodon.tokenization.CharTokenizer` |
+| `make`, `batch`, `acc` | `oglodon.data.algorithmic` (`make_example`, `AlgorithmicDataset`, `exact_match`) |
+| training loop | `oglodon.training.Trainer` |
+| causality / cache checks | `oglodon.validation.validate_model` and `tests/` |
+
+## Changes relative to the notebook (deliberate)
+
+* **Masking uses the dtype's most negative finite value instead of `-inf`**, so a fully masked row
+  (a padding query) gives a uniform softmax instead of NaN. This enabled padding masks.
+* **Padding masks and `position_ids`** (the notebook only handled equal-length batches).
+* **Dropout, optional learned positions, LayerNorm and GELU options**, all configurable.
+* **GPT-2-style initialisation** (`N(0, 0.02)`, residual projections scaled by `1/sqrt(2L)`).
+* **EOS handling** in generation (the notebook generated a fixed number of tokens), plus
+  temperature / top-k / top-p sampling.
+* **Trainer** averages loss over supervised tokens and weights gradient-accumulation micro-batches by
+  token count, so accumulation is equivalent to a larger batch.
+* The notebook's answer terminator `;` became a real `<eos>` token.
+
+## Design rules
+
+* Attention knows nothing about positions: RoPE is applied to `q`/`k` by the caller-supplied
+  `rope` tuple, so swapping position schemes does not touch attention code.
+* Attention maps are only returned when `output_attentions=True`; they are never stored on modules.
+* The Trainer only calls `model(**batch)` and reads `.loss`; any module with that contract trains.

oglodon_transformer-0.4.0/docs/cli.md

@@ -0,0 +1,101 @@
+# Command-line reference
+
+`oglodon <command> [options]` (also `python -m oglodon ...`). Run `oglodon <command> --help` for the authoritative
+list; this page explains what each command does.
+
+**Exit codes**: `0` success; `1` a check failed (`validate`, `verify`; with `--strict` also when something was
+skipped or unsupported); `2` usage or configuration error (bad flag, missing file, unusable settings).
+Errors print one `error: ...` line, not a traceback.
+
+## `oglodon train DATA... --out DIR`
+
+Train on text files. `DATA` is any mix of files, folders (searched recursively for text-like extensions such as
+`.txt .md .py .json .csv .tex`) and glob patterns.
+
+| option | default | meaning |
+|---|---|---|
+| `--out DIR` | required | checkpoint folder |
+| `--config FILE` | | YAML/JSON with `model:`, `data:`, `train:` sections; flags override it |
+| `--tokenizer {bpe,char}` | `bpe` | byte-level BPE trained on the training split, or one symbol per character |
+| `--vocab-size N` | 1024 | BPE vocabulary (the result can be smaller on a small corpus) |
+| `--val-fraction F` | 0.05 | held-out share of the tokens |
+| `--workers N` | 1 | parallel tokenization processes |
+| `--block-size N` | 128 | context length (`max_seq_len`) |
+| `--hidden-size N`, `--layers N`, `--heads N` | 128, 4, 8 | model size |
+| `--kv-heads N` | heads // 4 | key/value heads (GQA); `--kv-heads` equal to `--heads` is MHA |
+| `--steps N` | 1000 | optimizer steps |
+| `--batch-size N` | 16 | per process |
+| `--grad-accum N` | 1 | micro-batches per step |
+| `--lr F`, `--weight-decay F`, `--warmup N` | 1e-3, 0.1, 5% of steps | optimization |
+| `--seed N` | 0 | seeds data order and initialization |
+| `--precision {fp32,bf16,fp16}` | `fp32` | autocast; `fp16` needs CUDA |
+| `--eval-interval N`, `--save-interval N` | 10 evals/run, end only | evaluation and checkpoint cadence |
+| `--grad-checkpointing` | off | recompute activations: less memory, more compute |
+| `--compile` | off | `torch.compile` (first step is slow) |
+| `--device D` | cuda if available | `cpu`, `cuda`, `cuda:1`, ... |
+| `--resume` | off | continue the run in `--out` exactly where it stopped |
+| `--overwrite` | off | allow replacing a model already in `--out` |
+
+Without `--resume` or `--overwrite`, `train` refuses to touch a folder that already holds a model. Under
+`torchrun` it trains data-parallel automatically: see [Performance and scaling](performance-and-scaling.md).
+Details: [Training](training.md).
+
+## `oglodon generate CHECKPOINT`
+
+Stream text from a checkpoint made by `oglodon train` (or `save_pretrained(..., tokenizer=...)`).
+
+| option | default | meaning |
+|---|---|---|
+| `--prompt`, `-p TEXT` | stdin | text to continue; empty means unconditional |
+| `--max-new-tokens N` | 100 | must be below the model's context length |
+| `--temperature F` | 0.8 | `0` is greedy |
+| `--top-k N`, `--top-p F`, `--min-p F` | 40, off, off | sampling filters (`--top-k 0` disables) |
+| `--repetition-penalty F` | 1.0 | above 1 discourages repeats |
+| `--stop TEXT` | | stop after this text; repeatable |
+| `--seed N` | | reproducible sampling |
+| `--only-new` | off | do not echo the prompt |
+| `--no-stream` | off | print once at the end |
+
+Details: [Generation](generation.md).
+
+## `oglodon eval CHECKPOINT`
+
+At least one of: `--text FILE` (perplexity; `--block-size`, `--stride`, `--batch-size`), `--choices FILE.jsonl`
+(log-likelihood multiple choice), `--prompt P [P ...]` (greedy generation sanity check, `--max-new-tokens`).
+`--out FILE` writes a JSON report. Details: [Evaluation](evaluation.md).
+
+## `oglodon quantize CHECKPOINT --out DIR`
+
+Writes an int8 weight-only copy of a checkpoint (about 4x smaller linear weights; see [Quantization](quantization.md)).
+
+| flag | default | meaning |
+|---|---|---|
+| `--out DIR` | required | where the int8 checkpoint goes |
+| `--group-size N` | per channel | one scale per N input channels; must divide every quantized layer's input width |
+| `--include-lm-head` | off | also quantize the output matrix (only when embeddings are not tied) |
+| `--text FILE` | none | measure perplexity, KL and top-1 agreement on this text; without it drift is **not** measured |
+| `--block-size N` | model maximum | window length for that measurement |
+
+The result loads with `Oglodon.from_pretrained`, `oglodon generate` and `oglodon eval`.
+
+## `oglodon compare CONFIG [CONFIG ...]`
+
+Train architecture variants under identical conditions and report. One config with a `variants:` map, or several
+config files (one variant each; the first is the baseline). `--out FILE.json` (also writes `.md`), `--plot`
+(validation-loss curves as PNG, needs the `[viz]` extra), `--device`. Details: [Experiments](experiments.md).
+
+## `oglodon verify`
+
+Behavioural checks on any causal LM. Choose the model with exactly one of `--hf ID_OR_DIR`, `--hf-config FILE_OR_DIR`
+(random weights, offline), `--factory package.module:function`, `--checkpoint DIR`, `--config FILE`. Other options:
+`--cross-check`, `--categories`, `--exclude`, `--no-stress`, `--max-fp64-params`, `--device`, `--dtype`, `--seed`,
+`--vocab-size`, `--strict`, `--out`. Details: [verify](verify.md).
+
+## `oglodon validate CONFIG`
+
+Compare an Oglodon model against the independent float64 reference. `--checkpoint DIR`, `--categories`, `--device`,
+`--dtype`, `--seed`, `--strict`, `--out`. Details: [validation](validation.md).
+
+## `oglodon info`
+
+Prints versions, CUDA availability and bf16 support as JSON.