cogforge-engine 2.1.1__tar.gz → 2.1.3__tar.gz

{cogforge_engine-2.1.1 → cogforge_engine-2.1.3}/.gitignore

@@ -1,6 +1,7 @@
 # Python caches
 __pycache__/
 *.py[cod]
+.gitignore
 
 # Build folders (created by PyPI later)
 dist/
@@ -9,4 +10,7 @@ build/
 
 # Saved Model Weights
 *.npz
-*.pt
+*.pt
+
+project_review_report.md
+ROADMAP.md

{cogforge_engine-2.1.1 → cogforge_engine-2.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cogforge-engine
-Version: 2.1.1
+Version: 2.1.3
 Summary: A custom autograd engine and Transformer block built from scratch.
 Project-URL: Homepage, https://github.com/avikmjd2/cogforge
 Author-email: Avik Majumder <avikmjd2@gmail.com>
@@ -19,12 +19,15 @@ Description-Content-Type: text/markdown
 
 `cogforge` is a small, readable, educational deep learning framework. At its core is a `Tensor` that records every operation into a computation graph and backpropagates through it (micrograd-style), but unlike a toy autograd it scales up to real architectures: MLPs, RNNs, batch/layer normalization, multi-head self- and cross-attention, rotary position embeddings (RoPE), decoder-only GPTs, and a full encoder–decoder Seq2Seq transformer you can actually train and sample from.
 
-There is no C++, no PyTorch — just NumPy and explicit, hand-derived gradients. Optionally, the entire backend can be swapped to **CuPy** for GPU execution, or accelerated with **numexpr** on CPU, without changing any model code. The goal is still to *understand* every gradient that flows — speed is a bonus, not the point.
+There is no C++, no PyTorch — just NumPy and explicit, hand-derived gradients. Optionally, the entire backend can be swapped to **CuPy** for GPU execution, or accelerated with **numexpr** on CPU, without changing any model code. Every tensor also records the operation that produced it, and the built-in **Graphviz visualizer** can render the full computation graph — data, gradients, and ops — so you can literally *see* backpropagation. The goal is still to *understand* every gradient that flows — speed is a bonus, not the point.
+
+> **This release overhauls the loss API.** Seven overlapping cross-entropy variants have been unified into three functions covering every use case — with built-in **label smoothing**, optional padding masks everywhere, and one consistent shape/normalization convention. The superseded losses live on in `cogforge.legacy`. See [What's new](#whats-new).
 
 ---
 
 ## Table of contents
 
+- [What's new](#whats-new)
 - [Installation](#installation)
 - [Quick start](#quick-start)
 - [Backend: CPU, GPU, numexpr, and no-grad mode](#backend-cpu-gpu-numexpr-and-no-grad-mode)
@@ -42,12 +45,70 @@ There is no C++, no PyTorch — just NumPy and explicit, hand-derived gradients.
   - [Models](#models)
 - [Worked example: train a char-level GPT](#worked-example-train-a-char-level-gpt)
 - [Worked example: encoder–decoder Seq2Seq](#worked-example-encoderdecoder-seq2seq)
+- [Visualizing the computation graph](#visualizing-the-computation-graph)
 - [Gotchas](#gotchas)
 - [Roadmap](#roadmap)
 - [License](#license)
 
 ---
 
+## What's new
+
+This version consolidates the **loss API** into three functions with a single, consistent convention — and adds **label smoothing** on the road to a full translator.
+
+**Unified cross-entropy API (3 losses, one convention)**
+
+Seven overlapping cross-entropy variants (`softmax_cross_entropy_old`, `softmax_cross_entropy_masked`, `sparse_softmax_cross_entropy`, `sparse_softmax_cross_entropy_legacy`, `sparse_softmax_cross_entropy_index_legacy`, `cross_entropy_loss`, `cross_entropy_loss_masked`) are replaced by three that tile the whole space with no overlap — see [Losses](#losses):
+
+- **`Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask=None, eps=0.0)`** — the workhorse: logits in, integer labels, optional padding mask, optional **label smoothing** via `eps`. No one-hot matrix is ever built.
+- **`Tensor.softmax_cross_entropy(scores, targets, mask=None)`** — logits in, arbitrary target *distributions* (knowledge distillation, mixup, soft labels), optional mask.
+- **`Tensor.cross_entropy_from_probs(predictions, targets, mask=None)`** — for graphs that already end in `.softmax()`; prefer the logits-based losses when you have logits.
+
+All three now share the same rules, closing several long-standing traps:
+
+- **One shape convention.** Every loss accepts `(N, V)` *and* `(B, T, V)` logits (auto-flattened internally); masks may be `(N,)` or `(B, T)`.
+- **One normalization convention.** Every loss divides by the number of *real* (unmasked) tokens — no more `/B` vs `/B·T` vs `/n_real` drift between variants, which silently changed effective learning rates when switching losses.
+- **Masks are optional everywhere**, defaulting to "all positions are real".
+- **Exact log-probabilities.** Loss values are computed from `log-softmax` directly instead of `log(clip(p))`, so near-zero probabilities are no longer floored at `log(1e-15)`.
+
+**Label smoothing**
+
+`sparse_softmax_cross_entropy_index(..., eps=0.1)` trains against `(1−eps)` on the true class and `eps/(V−1)` spread uniformly over the rest — the standard regularizer for translation models — at zero extra memory: both loss and gradient are computed straight from the label indices. `eps=0.0` (default) is exact plain cross-entropy.
+
+**Performance**
+
+- The hot path (`eps=0`, no mask — e.g. char-level GPT training) keeps the fused, **numexpr-accelerated** in-place backward from the previous `sparse_softmax_cross_entropy`.
+- Masked/smoothed paths fold the mask, upstream gradient, and normalization into a single per-row coefficient — one fewer `(N, V)` temporary in every backward.
+
+**Legacy module**
+
+The superseded losses were **moved, not deleted**: they live in `cogforge.legacy` under `TensorLegacy`, unchanged, for reference and for reproducing old experiments. New code should not import them — note that they retain their old (inconsistent) normalization conventions.
+
+---
+
+### Previous release — autograd correctness pass
+
+This version focused on **autograd correctness and debuggability**. All fixes below were validated with finite-difference gradient checks and end-to-end training tests.
+
+**Gradient correctness fixes**
+
+- **Broadcast gradients fully generalized.** `+`, `-`, and `*` previously mishandled un-broadcasting when shapes broadcast along an *interior* size-1 dimension (e.g. `(B, 1, C) + (B, T, C)` — exactly the shape of a padding mask or any `keepdims` tensor): depending on shapes this either crashed or silently accumulated wrong gradients. All elementwise ops now route through a single, verified `Tensor.unbroadcast`, matching `@`.
+- **Indexing with repeated indices now accumulates.** `tensor[key]`'s backward used direct fancy-index assignment, which drops gradient contributions for duplicate indices due to NumPy write buffering. It now uses the backend-aware `scatter_add` (`np.add.at` on CPU, `cupyx.scatter_add` on GPU), so gathers with repeated indices — RoPE slicing, token lookups — backprop correctly.
+- **`sparse_softmax_cross_entropy` backward rewritten.** The gradient is now written straight into `scores.grad` as `(softmax − onehot) / N` without copying or mutating the cached softmax output — faster, less memory, and verified against numerical gradients (max error ~1e-4).
+- **`BatchNorm1D` is now safe to call multiple times before `backwards()`.** Backward state (`x_hat`, `std_inv`, the training flag) was previously stored on the module, so a second forward pass clobbered what the first pass's backward needed. Each call's backward closure now captures its own state — interleaved forwards produce bit-identical gradients to isolated ones.
+- **`NO_GRAD` fast paths are consistent everywhere:** no op builds a graph node it's about to discard.
+
+**Generation fixes**
+
+- **`Seq2Seq.generate` handles per-sequence EOS.** Previously the loop only stopped if *every* row in the batch emitted `eos_id` on the *same step*, and finished rows kept sampling. Each row is now frozen to `pad_id` the moment it emits EOS, and the loop exits once all rows are done.
+
+**New: graph visualization (`cogforge.utils`)**
+
+- Every `Tensor` now carries a human-readable `.op` label (`"+"`, `"MATMUL"`, `"softmax"`, `"layernorm"`, …) recording the operation that produced it.
+- `draw_graph(tensor)` renders the full computation graph via Graphviz — one record node per tensor showing its op, shape, data, and gradient. See [Visualizing the computation graph](#visualizing-the-computation-graph).
+
+---
+
 ## Installation
 
 ```bash
@@ -59,15 +120,18 @@ Requires Python 3.8+ and NumPy — that's the only hard dependency. Two optional
 ```bash
 pip install cupy-cuda12x   # GPU backend (pick the build matching your CUDA version)
 pip install numexpr        # multi-threaded CPU element-wise ops
+pip install graphviz       # computation-graph visualization (also needs the Graphviz system binaries)
 ```
 
-The package is organized into three modules:
+The package is organized into four modules:
 
 | Module | Contains |
 | --- | --- |
 | `cogforge.backend` | The swappable array backend: NumPy ↔ CuPy switching, numexpr flag, global no-grad flag. |
 | `cogforge.app` | The autograd engine (`Tensor`) and every building block — layers, optimizers, losses, normalization, attention, positional encodings. |
 | `cogforge.models` | Ready-to-use models: `GPTV1`, `GPT2`, `Seq2Seq`. |
+| `cogforge.utils` | Computation-graph tracing and Graphviz rendering (`trace_graph`, `draw_graph`). |
+| `cogforge.legacy` | `TensorLegacy` — the superseded cross-entropy variants, kept unchanged for reference and reproducing old experiments. Don't use in new code. |
 
 ```python
 from cogforge.app import Tensor, Linear, Adam, MultiHeadAttention   # building blocks
@@ -158,9 +222,12 @@ Tensor(array, children=(), requires_grad=True, typed="compressed")
 | `children` | Parent tensors in the graph (set internally by ops; you rarely pass this). |
 | `requires_grad` | Reserved flag (currently informational). |
 | `typed` | `"compressed"` → `float32` (default), anything else → `float64`. |
+| `op` | Human-readable label of the operation that produced this tensor (`"+"`, `"MATMUL"`, `"softmax"`, …). Set internally by every op; used by the graph visualizer and handy when debugging. |
 
 Gradients **accumulate** into `.grad`. Always zero them between optimization steps (the optimizers do this for you via `zero_grad()`). When global no-grad mode is on, `.grad` is `None` and no graph is recorded.
 
+> **The graph is single-use.** `backwards()` frees the graph as it goes (clearing each node's children and backward closure) to release memory eagerly. Run one forward → one backward per step, and if you want to [visualize the graph](#visualizing-the-computation-graph), draw it **before** calling `backwards()`.
+
 ---
 
 ## API reference
@@ -171,9 +238,9 @@ Gradients **accumulate** into `.grad`. Always zero them between optimization ste
 
 | Operation | Notes |
 | --- | --- |
-| `a + b`, `a - b`, `a * b` | Elementwise, with broadcasting support. |
+| `a + b`, `a - b`, `a * b` | Elementwise, with full broadcasting support — gradients are correctly un-broadcast for leading *and* interior size-1 dimensions. |
 | `a @ b` | Batched matmul; gradients are correctly un-broadcast. |
-| `a[key]` | Indexing/slicing. |
+| `a[key]` | Indexing/slicing. Backward uses backend-aware scatter-add, so **repeated indices accumulate gradients correctly** on CPU and GPU. |
 | `Tensor.cat(tensors, axis=-1)` | Classmethod. Concatenates a tuple of tensors along `axis`; backward splits the gradient back to each parent. (Used internally by RoPE.) |
 | `.relu()` | |
 | `.sigmoid()` | |
@@ -191,7 +258,7 @@ Gradients **accumulate** into `.grad`. Always zero them between optimization ste
 
 | Method | Notes |
 | --- | --- |
-| `.backwards()` | **Primary.** Iterative topological sort — safe for deep/long graphs. |
+| `.backwards()` | **Primary.** Iterative topological sort — safe for deep/long graphs. Frees the graph as it runs (single-use; see the note under [Core concept](#core-concept-the-tensor)). |
 | `.backwards_recursive()` | Legacy recursive version; can hit Python's recursion limit on long sequences. Prefer `.backwards()`. |
 
 **Static helper**
@@ -202,20 +269,23 @@ Gradients **accumulate** into `.grad`. Always zero them between optimization ste
 
 ### Losses
 
-All losses are **classmethods** on `Tensor` and return a scalar loss tensor you call `.backwards()` on. Mind the distinction between losses that take **probabilities** and losses that take **raw logits** — this is the most common mistake.
+All losses are **classmethods** on `Tensor` and return a scalar loss tensor you call `.backwards()` on. There are exactly **three**, and they share one convention:
+
+- **Shapes:** logits/probs may be `(N, V)` or `(B, T, V)` — 3-D input is auto-flattened. Masks may be `(N,)` or `(B, T)`.
+- **Masks:** optional everywhere (`mask=None` treats all positions as real); `1` = real token, `0` = padding. Padded positions contribute zero loss *and* exactly zero gradient.
+- **Normalization:** always by the number of real tokens (all positions when unmasked).
+
+Pick by what your **target** looks like and what your **input** is:
 
-| Loss | Input expectation | Use when |
-| --- | --- | --- |
-| `Tensor.softmax_cross_entropy(scores, targets)` | `scores` are **raw logits**, `targets` one-hot. Softmax fused inside (stable). Works for 2-D `(B,V)` and 3-D `(B,T,V)`. | Standard classification / LM. **Recommended.** |
-| `Tensor.sparse_softmax_cross_entropy(scores, target_ids)` | `scores` raw logits `(B,T,V)`, `target_ids` integers `(B,T)`. | Language modeling — skips building one-hot targets. |
-| `Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask)` | Raw logits + integer labels + per-token mask (1 = real, 0 = pad). | Padded LM / seq2seq batches. **Recommended for Seq2Seq training.** |
-| `Tensor.softmax_cross_entropy_masked(scores, targets, mask)` | Logits + one-hot targets + per-row mask. | Padded batches, fused softmax. |
-| `Tensor.cross_entropy_loss(predictions, targets)` | `predictions` are **probabilities** (call `.softmax()` first), `targets` one-hot. | You already have a softmax in your graph. |
-| `Tensor.cross_entropy_loss_masked(predictions, targets, mask)` | Probabilities + per-row mask. | Padded batches without fused softmax. |
+| Loss | Input | Target | Use when |
+| --- | --- | --- | --- |
+| `Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask=None, eps=0.0)` | **raw logits** | integer class ids | **The workhorse.** Classification, LM, seq2seq. `eps>0` enables label smoothing: `1−eps` on the true class, `eps/(V−1)` on the rest — computed straight from indices, no one-hot built. `eps=0.1` is the standard for translation. |
+| `Tensor.softmax_cross_entropy(scores, targets, mask=None)` | **raw logits** | full distribution `(…, V)`, rows sum to 1 | Targets that aren't expressible as an index: knowledge distillation against a teacher's output, mixup, annotator vote distributions. |
+| `Tensor.cross_entropy_from_probs(predictions, targets, mask=None)` | **probabilities** (e.g. output of `.softmax()`) | one-hot or distribution | Your graph already ends in a softmax. Prefer the logits-based losses when you have logits — their gradients are exact and bounded, this one saturates via clipping as probs → 0. |
 
-> ℹ️ The `softmax_*` and `sparse_softmax_*` variants apply softmax internally — feed them **raw logits**. `cross_entropy_loss*` is the opposite — it expects probabilities.
+> ℹ️ The first two apply softmax internally — feed them **raw logits**. `cross_entropy_from_probs` is the opposite — it expects probabilities. Mixing these up silently trains the wrong thing.
 >
-> `sparse_softmax_cross_entropy_legacy` and `softmax_cross_entropy_old` are kept for reference; prefer the current versions.
+> The previous generation of losses (`softmax_cross_entropy_masked`, `sparse_softmax_cross_entropy`, `cross_entropy_loss`, `cross_entropy_loss_masked`, and the `_old`/`_legacy` variants) now lives in `cogforge.legacy` under `TensorLegacy`, unchanged. They keep their old, mutually inconsistent normalizations (`/B` vs `/B·T` vs `/n_real`) — use them only to reproduce old experiments, and don't mix them with the current losses in one training run.
 
 ---
 
@@ -231,7 +301,7 @@ Lookup table. Call with an integer index array; backward scatters gradients corr
 Normalizes over the last dimension. Learnable `gamma`/`beta`, full hand-derived backward. `.parameters()` → `[gamma, beta]`.
 
 #### `BatchNorm1D(dim, eps=1e-5, momentum=0.1)`
-Normalizes over the batch (and time, for 3-D input). Tracks `running_mean`/`running_var` for inference. Toggle `.training = True/False`. Learnable `gamma`/`beta`.
+Normalizes over the batch (and time, for 3-D input). Tracks `running_mean`/`running_var` for inference. Toggle `.training = True/False`. Learnable `gamma`/`beta`. Each forward call captures its own backward state, so calling the layer multiple times before `backwards()` (e.g. gradient accumulation, shared modules) yields correct gradients for every call.
 
 #### `FeedForward(dmodel, dff=None)`
 Position-wise MLP: `Linear → ReLU → Dropout(0.15) → Linear`. `dff` defaults to `4 * dmodel`. Dropout is active only when called with `is_training=True` (transformer blocks handle this for you via their train/infer state).
@@ -380,7 +450,7 @@ A full **encoder–decoder transformer** (the original *Attention Is All You Nee
 | `model(enc_idx, dec_idx)` | Teacher-forced forward. `enc_idx`: source `(B, T_enc)`; `dec_idx`: shifted target starting with `<SOS>`, `(B, T_dec)`. Returns logits `(B, T_dec, dec_vocab)`. |
 | `model.encode(enc_idx)` | Run the encoder once; returns `(x_enc, enc_pad_mask)` for reuse across decode steps. |
 | `model.decode_step(dec_idx, x_enc, enc_pad)` | Decoder forward against a fixed encoder output. |
-| `model.generate(enc_idx, sos_id, eos_id=None, max_new=50, temperature=1.0, top_k=None)` | Encodes once, then autoregressively decodes from `<SOS>`; stops early when every sequence in the batch emits `eos_id`. |
+| `model.generate(enc_idx, sos_id, eos_id=None, max_new=50, temperature=1.0, top_k=None)` | Encodes once, then autoregressively decodes from `<SOS>`. Tracks completion **per sequence**: the moment a row emits `eos_id` it is frozen and padded with `pad_id` for the remaining steps, and decoding stops early once every row has finished. |
 | `model.train()` / `model.infer()` | Toggle dropout across all encoder and decoder blocks. |
 | `Seq2Seq.make_pad_mask(idx, pad_id)` | Static helper: `(B, T)` ints → `(B, 1, 1, T)` boolean mask, `True` at padding. |
 
@@ -419,7 +489,7 @@ def get_batch(bs=32):
 for step in range(2000):
     x, y   = get_batch()
     logits = model(x)                                  # (B, T, vocab)
-    loss   = Tensor.sparse_softmax_cross_entropy(logits, y)
+    loss   = Tensor.sparse_softmax_cross_entropy_index(logits, y)
 
     opt.zero_grad()
     loss.backwards()
@@ -472,25 +542,56 @@ for step in range(num_steps):
     dec_in, labels = tgt[:, :-1], tgt[:, 1:]
     logits = model(src, dec_in)            # (B, T_dec, vocab)
 
-    mask = (labels != PAD).astype(np.float32)
-    loss = Tensor.sparse_softmax_cross_entropy_index(logits, labels, mask)
+    mask = (labels != PAD).astype(np.float32)     # (B, T_dec): 1 = real, 0 = pad
+    loss = Tensor.sparse_softmax_cross_entropy_index(logits, labels,
+                                                     mask=mask, eps=0.1)  # label smoothing
 
     opt.zero_grad()
     loss.backwards()
     opt.clip_grads(1.0)
     opt.step()
 
-# inference: encode once, decode token by token, stop on EOS
+# inference: encode once, decode token by token, stop on EOS (per sequence)
 out = model.generate(src, sos_id=SOS, eos_id=EOS, max_new=32,
                      temperature=1.0, top_k=5)
 ```
 
 ---
 
+## Visualizing the computation graph
+
+`cogforge.utils` renders the autograd graph with Graphviz — every node shows the op that produced it, its shape, its data, and its gradient. It's the fastest way to *see* what your model is doing and to debug shape or gradient-flow issues.
+
+```python
+import numpy as np
+from cogforge.app import Tensor, Linear
+from cogforge.utils import draw_graph
+
+x   = Tensor(np.random.randn(2, 4))
+lin = Linear(4, 3)
+out = lin(x).relu().softmax()
+
+dot = draw_graph(out, rankdir='LR')   # draw BEFORE backwards() — the graph is freed by it
+dot.render('graph', view=True)        # writes graph.svg and opens it
+```
+
+| Function | Purpose |
+| --- | --- |
+| `trace_graph(root_tensor)` | Walks the graph from a root tensor; returns `(nodes, edges)`. |
+| `draw_graph(root_tensor, max_char=50, format='svg', rankdir='LR')` | Builds a Graphviz `Digraph`. `max_char` truncates long data/grad strings; `format` is any Graphviz output format (`svg`, `png`, `pdf`); `rankdir='TB'` for top-to-bottom layout. |
+
+Nodes produced by a named op render white with the op label; raw leaf tensors (parameters, inputs) render light blue. Requires the `graphviz` Python package **and** the Graphviz system binaries (`apt install graphviz` / `brew install graphviz`).
+
+Two practical notes: call `draw_graph` **before** `backwards()`, since the backward pass frees the graph as it runs (afterwards you'll see a single orphan node); and keep the visualized graph small — a full transformer forward produces hundreds of nodes, so draw a single block or a toy input rather than a whole training step.
+
+---
+
 ## Gotchas
 
 - **It's `backwards()`, not `backward()`.** The backward pass method has a trailing `s`.
-- **Logits vs. probabilities.** `softmax_cross_entropy` / `sparse_softmax_cross_entropy*` fuse the softmax internally — feed them **raw logits**. `cross_entropy_loss*` expects **probabilities**. Mixing these up silently trains the wrong thing.
+- **Logits vs. probabilities.** `softmax_cross_entropy` and `sparse_softmax_cross_entropy_index` fuse the softmax internally — feed them **raw logits**. `cross_entropy_from_probs` expects **probabilities**. Mixing these up silently trains the wrong thing.
+- **Label smoothing raises the loss floor.** With `eps > 0` the minimum achievable loss is no longer 0 (a perfect model still pays the smoothing term), so judge training by token accuracy or validation metrics, not by how close the raw loss gets to zero — and don't compare loss values across different `eps` settings.
+- **Legacy losses normalize differently.** `cogforge.legacy.TensorLegacy` keeps the old `/B` / `/B·T` conventions; the current losses always divide by real-token count. Swapping one for the other rescales gradients — retune the learning rate if you migrate an old script.
 - **Gradients accumulate.** Call `optimizer.zero_grad()` every step (or `p.grad[...] = 0`), or gradients pile up across iterations.
 - **Switch the backend before building the model.** `use_gpu(True)` after construction leaves your parameters stranded on the CPU while new activations land on the GPU.
 - **RoPE dim is per-head.** `RotatoryPositionalEncoding` takes `d_model // n_heads` (which must be even), not `d_model`. The models handle this internally — it only matters if you wire blocks up yourself.
@@ -498,13 +599,26 @@ out = model.generate(src, sos_id=SOS, eos_id=EOS, max_new=32,
 - **`Sequential.parameters()` is shallow** — see the note under [Containers](#containers). For attention/feed-forward/transformer stacks, gather parameters through each module's own `.parameters()` (as the models' `parameters()` methods do).
 - **RNNs operate on lists**, not a single `(B, T, C)` tensor — pass a list of per-timestep tensors.
 - **Don't dedupe tied parameters yourself.** `Seq2Seq.parameters()` already deduplicates shared tensors by identity, so the tied embedding is only updated once per step.
+- **The graph is single-use.** `backwards()` frees children and backward closures as it runs. One forward → one backward. Calling `backwards()` twice on the same graph is a silent no-op the second time, and `draw_graph` must be called before, not after.
+- **Visualize small graphs.** `draw_graph` on a full model forward will produce an unreadable diagram with hundreds of nodes; visualize a single layer or block instead.
 
 ---
 
 ## Roadmap
 
-Shipped since the last release:
+Shipped in this release:
+
+- ✅ **Unified loss API** — three cross-entropy losses covering index targets, distribution targets, and probability inputs, with one shape and normalization convention (see [What's new](#whats-new))
+- ✅ **Label smoothing** (`eps`) in `sparse_softmax_cross_entropy_index`, computed from indices with zero extra memory
+- ✅ Optional padding masks and `(B, T, V)` auto-flattening in every loss
+- ✅ `cogforge.legacy` module (`TensorLegacy`) preserving the superseded losses for reproducibility
+
+Shipped previously:
 
+- ✅ **Autograd correctness pass** — all gradients verified against finite-difference checks; broadcast, indexing, sparse cross-entropy, and BatchNorm backward bugs fixed
+- ✅ Computation-graph visualization via Graphviz (`cogforge.utils`), with per-tensor `op` labels
+- ✅ Per-sequence EOS handling in `Seq2Seq.generate` (finished rows freeze to `pad_id`)
+- ✅ Eager graph freeing in `backwards()` for lower peak memory
 - ✅ RoPE (rotary position embeddings), usable in GPT and at every attention site of the Seq2Seq model
 - ✅ Full encoder–decoder transformer (`Seq2Seq`) with cross-attention and padding masks
 - ✅ Weight tying between embeddings and the output head
@@ -514,14 +628,19 @@ Shipped since the last release:
 
 Planned / under consideration:
 
+- Beam search decoding for `Seq2Seq` (length-normalized, per-sequence finished pool)
+- Lazy Gradient Allocation (LGA) for lower peak memory on long sequences
+- Cosine LR function as part of UTILS module
+- Weight decay / AdamW optimizer
+- Checkpoint save/load (`state_dict` / `load_state_dict`) for the transformer models, including optimizer state
+- `no_grad()` context manager and a shared `Module` base class
 - KV cache for faster generation
 - SwiGLU feed-forward and RMSNorm
 - RoPE length interpolation
 - Linear-attention block (as a study in the recall-vs-cost tradeoff)
-- Checkpoint save/load for the transformer models
 
 ---
 
 ## License
 
-MIT License. See [LICENSE](LICENSE) for details.
+APACHE License. See [LICENSE](LICENSE) for details.