cogforge-engine 2.1.2__tar.gz → 2.1.3__tar.gz

{cogforge_engine-2.1.2 → cogforge_engine-2.1.3}/.gitignore

@@ -1,6 +1,7 @@
 # Python caches
 __pycache__/
 *.py[cod]
+.gitignore
 
 # Build folders (created by PyPI later)
 dist/
@@ -9,4 +10,7 @@ build/
 
 # Saved Model Weights
 *.npz
-*.pt
+*.pt
+
+project_review_report.md
+ROADMAP.md

{cogforge_engine-2.1.2 → cogforge_engine-2.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cogforge-engine
-Version: 2.1.2
+Version: 2.1.3
 Summary: A custom autograd engine and Transformer block built from scratch.
 Project-URL: Homepage, https://github.com/avikmjd2/cogforge
 Author-email: Avik Majumder <avikmjd2@gmail.com>
@@ -21,7 +21,7 @@ Description-Content-Type: text/markdown
 
 There is no C++, no PyTorch — just NumPy and explicit, hand-derived gradients. Optionally, the entire backend can be swapped to **CuPy** for GPU execution, or accelerated with **numexpr** on CPU, without changing any model code. Every tensor also records the operation that produced it, and the built-in **Graphviz visualizer** can render the full computation graph — data, gradients, and ops — so you can literally *see* backpropagation. The goal is still to *understand* every gradient that flows — speed is a bonus, not the point.
 
-> **This release is a correctness pass.** Every gradient in the engine has now been verified against finite-difference numerical checks, and several subtle autograd bugs were found and fixed — see [What's new](#whats-new).
+> **This release overhauls the loss API.** Seven overlapping cross-entropy variants have been unified into three functions covering every use case — with built-in **label smoothing**, optional padding masks everywhere, and one consistent shape/normalization convention. The superseded losses live on in `cogforge.legacy`. See [What's new](#whats-new).
 
 ---
 
@@ -54,7 +54,41 @@ There is no C++, no PyTorch — just NumPy and explicit, hand-derived gradients.
 
 ## What's new
 
-This version focuses on **autograd correctness and debuggability**. All fixes below were validated with finite-difference gradient checks and end-to-end training tests.
+This version consolidates the **loss API** into three functions with a single, consistent convention — and adds **label smoothing** on the road to a full translator.
+
+**Unified cross-entropy API (3 losses, one convention)**
+
+Seven overlapping cross-entropy variants (`softmax_cross_entropy_old`, `softmax_cross_entropy_masked`, `sparse_softmax_cross_entropy`, `sparse_softmax_cross_entropy_legacy`, `sparse_softmax_cross_entropy_index_legacy`, `cross_entropy_loss`, `cross_entropy_loss_masked`) are replaced by three that tile the whole space with no overlap — see [Losses](#losses):
+
+- **`Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask=None, eps=0.0)`** — the workhorse: logits in, integer labels, optional padding mask, optional **label smoothing** via `eps`. No one-hot matrix is ever built.
+- **`Tensor.softmax_cross_entropy(scores, targets, mask=None)`** — logits in, arbitrary target *distributions* (knowledge distillation, mixup, soft labels), optional mask.
+- **`Tensor.cross_entropy_from_probs(predictions, targets, mask=None)`** — for graphs that already end in `.softmax()`; prefer the logits-based losses when you have logits.
+
+All three now share the same rules, closing several long-standing traps:
+
+- **One shape convention.** Every loss accepts `(N, V)` *and* `(B, T, V)` logits (auto-flattened internally); masks may be `(N,)` or `(B, T)`.
+- **One normalization convention.** Every loss divides by the number of *real* (unmasked) tokens — no more `/B` vs `/B·T` vs `/n_real` drift between variants, which silently changed effective learning rates when switching losses.
+- **Masks are optional everywhere**, defaulting to "all positions are real".
+- **Exact log-probabilities.** Loss values are computed from `log-softmax` directly instead of `log(clip(p))`, so near-zero probabilities are no longer floored at `log(1e-15)`.
+
+**Label smoothing**
+
+`sparse_softmax_cross_entropy_index(..., eps=0.1)` trains against `(1−eps)` on the true class and `eps/(V−1)` spread uniformly over the rest — the standard regularizer for translation models — at zero extra memory: both loss and gradient are computed straight from the label indices. `eps=0.0` (default) is exact plain cross-entropy.
+
+**Performance**
+
+- The hot path (`eps=0`, no mask — e.g. char-level GPT training) keeps the fused, **numexpr-accelerated** in-place backward from the previous `sparse_softmax_cross_entropy`.
+- Masked/smoothed paths fold the mask, upstream gradient, and normalization into a single per-row coefficient — one fewer `(N, V)` temporary in every backward.
+
+**Legacy module**
+
+The superseded losses were **moved, not deleted**: they live in `cogforge.legacy` under `TensorLegacy`, unchanged, for reference and for reproducing old experiments. New code should not import them — note that they retain their old (inconsistent) normalization conventions.
+
+---
+
+### Previous release — autograd correctness pass
+
+This version focused on **autograd correctness and debuggability**. All fixes below were validated with finite-difference gradient checks and end-to-end training tests.
 
 **Gradient correctness fixes**
 
@@ -97,6 +131,7 @@ The package is organized into four modules:
 | `cogforge.app` | The autograd engine (`Tensor`) and every building block — layers, optimizers, losses, normalization, attention, positional encodings. |
 | `cogforge.models` | Ready-to-use models: `GPTV1`, `GPT2`, `Seq2Seq`. |
 | `cogforge.utils` | Computation-graph tracing and Graphviz rendering (`trace_graph`, `draw_graph`). |
+| `cogforge.legacy` | `TensorLegacy` — the superseded cross-entropy variants, kept unchanged for reference and reproducing old experiments. Don't use in new code. |
 
 ```python
 from cogforge.app import Tensor, Linear, Adam, MultiHeadAttention   # building blocks
@@ -234,20 +269,23 @@ Gradients **accumulate** into `.grad`. Always zero them between optimization ste
 
 ### Losses
 
-All losses are **classmethods** on `Tensor` and return a scalar loss tensor you call `.backwards()` on. Mind the distinction between losses that take **probabilities** and losses that take **raw logits** — this is the most common mistake.
+All losses are **classmethods** on `Tensor` and return a scalar loss tensor you call `.backwards()` on. There are exactly **three**, and they share one convention:
 
-| Loss | Input expectation | Use when |
-| --- | --- | --- |
-| `Tensor.softmax_cross_entropy(scores, targets)` | `scores` are **raw logits**, `targets` one-hot. Softmax fused inside (stable). Works for 2-D `(B,V)` and 3-D `(B,T,V)`. | Standard classification / LM. **Recommended.** |
-| `Tensor.sparse_softmax_cross_entropy(scores, target_ids)` | `scores` raw logits `(B,T,V)`, `target_ids` integers `(B,T)`. | Language modeling — skips building one-hot targets. |
-| `Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask)` | Raw logits + integer labels + per-token mask (1 = real, 0 = pad). | Padded LM / seq2seq batches. **Recommended for Seq2Seq training.** |
-| `Tensor.softmax_cross_entropy_masked(scores, targets, mask)` | Logits + one-hot targets + per-row mask. | Padded batches, fused softmax. |
-| `Tensor.cross_entropy_loss(predictions, targets)` | `predictions` are **probabilities** (call `.softmax()` first), `targets` one-hot. | You already have a softmax in your graph. |
-| `Tensor.cross_entropy_loss_masked(predictions, targets, mask)` | Probabilities + per-row mask. | Padded batches without fused softmax. |
+- **Shapes:** logits/probs may be `(N, V)` or `(B, T, V)` — 3-D input is auto-flattened. Masks may be `(N,)` or `(B, T)`.
+- **Masks:** optional everywhere (`mask=None` treats all positions as real); `1` = real token, `0` = padding. Padded positions contribute zero loss *and* exactly zero gradient.
+- **Normalization:** always by the number of real tokens (all positions when unmasked).
 
-> ℹ️ The `softmax_*` and `sparse_softmax_*` variants apply softmax internally — feed them **raw logits**. `cross_entropy_loss*` is the opposite — it expects probabilities.
+Pick by what your **target** looks like and what your **input** is:
+
+| Loss | Input | Target | Use when |
+| --- | --- | --- | --- |
+| `Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask=None, eps=0.0)` | **raw logits** | integer class ids | **The workhorse.** Classification, LM, seq2seq. `eps>0` enables label smoothing: `1−eps` on the true class, `eps/(V−1)` on the rest — computed straight from indices, no one-hot built. `eps=0.1` is the standard for translation. |
+| `Tensor.softmax_cross_entropy(scores, targets, mask=None)` | **raw logits** | full distribution `(…, V)`, rows sum to 1 | Targets that aren't expressible as an index: knowledge distillation against a teacher's output, mixup, annotator vote distributions. |
+| `Tensor.cross_entropy_from_probs(predictions, targets, mask=None)` | **probabilities** (e.g. output of `.softmax()`) | one-hot or distribution | Your graph already ends in a softmax. Prefer the logits-based losses when you have logits — their gradients are exact and bounded, this one saturates via clipping as probs → 0. |
+
+> ℹ️ The first two apply softmax internally — feed them **raw logits**. `cross_entropy_from_probs` is the opposite — it expects probabilities. Mixing these up silently trains the wrong thing.
 >
-> `sparse_softmax_cross_entropy_legacy` and `softmax_cross_entropy_old` are kept for reference; prefer the current versions.
+> The previous generation of losses (`softmax_cross_entropy_masked`, `sparse_softmax_cross_entropy`, `cross_entropy_loss`, `cross_entropy_loss_masked`, and the `_old`/`_legacy` variants) now lives in `cogforge.legacy` under `TensorLegacy`, unchanged. They keep their old, mutually inconsistent normalizations (`/B` vs `/B·T` vs `/n_real`) — use them only to reproduce old experiments, and don't mix them with the current losses in one training run.
 
 ---
 
@@ -451,7 +489,7 @@ def get_batch(bs=32):
 for step in range(2000):
     x, y   = get_batch()
     logits = model(x)                                  # (B, T, vocab)
-    loss   = Tensor.sparse_softmax_cross_entropy(logits, y)
+    loss   = Tensor.sparse_softmax_cross_entropy_index(logits, y)
 
     opt.zero_grad()
     loss.backwards()
@@ -504,8 +542,9 @@ for step in range(num_steps):
     dec_in, labels = tgt[:, :-1], tgt[:, 1:]
     logits = model(src, dec_in)            # (B, T_dec, vocab)
 
-    mask = (labels != PAD).astype(np.float32)
-    loss = Tensor.sparse_softmax_cross_entropy_index(logits, labels, mask)
+    mask = (labels != PAD).astype(np.float32)     # (B, T_dec): 1 = real, 0 = pad
+    loss = Tensor.sparse_softmax_cross_entropy_index(logits, labels,
+                                                     mask=mask, eps=0.1)  # label smoothing
 
     opt.zero_grad()
     loss.backwards()
@@ -550,7 +589,9 @@ Two practical notes: call `draw_graph` **before** `backwards()`, since the backw
 ## Gotchas
 
 - **It's `backwards()`, not `backward()`.** The backward pass method has a trailing `s`.
-- **Logits vs. probabilities.** `softmax_cross_entropy` / `sparse_softmax_cross_entropy*` fuse the softmax internally — feed them **raw logits**. `cross_entropy_loss*` expects **probabilities**. Mixing these up silently trains the wrong thing.
+- **Logits vs. probabilities.** `softmax_cross_entropy` and `sparse_softmax_cross_entropy_index` fuse the softmax internally — feed them **raw logits**. `cross_entropy_from_probs` expects **probabilities**. Mixing these up silently trains the wrong thing.
+- **Label smoothing raises the loss floor.** With `eps > 0` the minimum achievable loss is no longer 0 (a perfect model still pays the smoothing term), so judge training by token accuracy or validation metrics, not by how close the raw loss gets to zero — and don't compare loss values across different `eps` settings.
+- **Legacy losses normalize differently.** `cogforge.legacy.TensorLegacy` keeps the old `/B` / `/B·T` conventions; the current losses always divide by real-token count. Swapping one for the other rescales gradients — retune the learning rate if you migrate an old script.
 - **Gradients accumulate.** Call `optimizer.zero_grad()` every step (or `p.grad[...] = 0`), or gradients pile up across iterations.
 - **Switch the backend before building the model.** `use_gpu(True)` after construction leaves your parameters stranded on the CPU while new activations land on the GPU.
 - **RoPE dim is per-head.** `RotatoryPositionalEncoding` takes `d_model // n_heads` (which must be even), not `d_model`. The models handle this internally — it only matters if you wire blocks up yourself.
@@ -567,13 +608,17 @@ Two practical notes: call `draw_graph` **before** `backwards()`, since the backw
 
 Shipped in this release:
 
-- ✅ **Autograd correctness pass** — all gradients verified against finite-difference checks; broadcast, indexing, sparse cross-entropy, and BatchNorm backward bugs fixed (see [What's new](#whats-new))
-- ✅ Computation-graph visualization via Graphviz (`cogforge.utils`), with per-tensor `op` labels
-- ✅ Per-sequence EOS handling in `Seq2Seq.generate` (finished rows freeze to `pad_id`)
-- ✅ Eager graph freeing in `backwards()` for lower peak memory
+- ✅ **Unified loss API** — three cross-entropy losses covering index targets, distribution targets, and probability inputs, with one shape and normalization convention (see [What's new](#whats-new))
+- ✅ **Label smoothing** (`eps`) in `sparse_softmax_cross_entropy_index`, computed from indices with zero extra memory
+- ✅ Optional padding masks and `(B, T, V)` auto-flattening in every loss
+- ✅ `cogforge.legacy` module (`TensorLegacy`) preserving the superseded losses for reproducibility
 
 Shipped previously:
 
+- ✅ **Autograd correctness pass** — all gradients verified against finite-difference checks; broadcast, indexing, sparse cross-entropy, and BatchNorm backward bugs fixed
+- ✅ Computation-graph visualization via Graphviz (`cogforge.utils`), with per-tensor `op` labels
+- ✅ Per-sequence EOS handling in `Seq2Seq.generate` (finished rows freeze to `pad_id`)
+- ✅ Eager graph freeing in `backwards()` for lower peak memory
 - ✅ RoPE (rotary position embeddings), usable in GPT and at every attention site of the Seq2Seq model
 - ✅ Full encoder–decoder transformer (`Seq2Seq`) with cross-attention and padding masks
 - ✅ Weight tying between embeddings and the output head
@@ -583,6 +628,7 @@ Shipped previously:
 
 Planned / under consideration:
 
+- Beam search decoding for `Seq2Seq` (length-normalized, per-sequence finished pool)
 - Lazy Gradient Allocation (LGA) for lower peak memory on long sequences
 - Cosine LR function as part of UTILS module
 - Weight decay / AdamW optimizer

{cogforge_engine-2.1.2 → cogforge_engine-2.1.3}/README.md

@@ -6,7 +6,7 @@
 
 There is no C++, no PyTorch — just NumPy and explicit, hand-derived gradients. Optionally, the entire backend can be swapped to **CuPy** for GPU execution, or accelerated with **numexpr** on CPU, without changing any model code. Every tensor also records the operation that produced it, and the built-in **Graphviz visualizer** can render the full computation graph — data, gradients, and ops — so you can literally *see* backpropagation. The goal is still to *understand* every gradient that flows — speed is a bonus, not the point.
 
-> **This release is a correctness pass.** Every gradient in the engine has now been verified against finite-difference numerical checks, and several subtle autograd bugs were found and fixed — see [What's new](#whats-new).
+> **This release overhauls the loss API.** Seven overlapping cross-entropy variants have been unified into three functions covering every use case — with built-in **label smoothing**, optional padding masks everywhere, and one consistent shape/normalization convention. The superseded losses live on in `cogforge.legacy`. See [What's new](#whats-new).
 
 ---
 
@@ -39,7 +39,41 @@ There is no C++, no PyTorch — just NumPy and explicit, hand-derived gradients.
 
 ## What's new
 
-This version focuses on **autograd correctness and debuggability**. All fixes below were validated with finite-difference gradient checks and end-to-end training tests.
+This version consolidates the **loss API** into three functions with a single, consistent convention — and adds **label smoothing** on the road to a full translator.
+
+**Unified cross-entropy API (3 losses, one convention)**
+
+Seven overlapping cross-entropy variants (`softmax_cross_entropy_old`, `softmax_cross_entropy_masked`, `sparse_softmax_cross_entropy`, `sparse_softmax_cross_entropy_legacy`, `sparse_softmax_cross_entropy_index_legacy`, `cross_entropy_loss`, `cross_entropy_loss_masked`) are replaced by three that tile the whole space with no overlap — see [Losses](#losses):
+
+- **`Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask=None, eps=0.0)`** — the workhorse: logits in, integer labels, optional padding mask, optional **label smoothing** via `eps`. No one-hot matrix is ever built.
+- **`Tensor.softmax_cross_entropy(scores, targets, mask=None)`** — logits in, arbitrary target *distributions* (knowledge distillation, mixup, soft labels), optional mask.
+- **`Tensor.cross_entropy_from_probs(predictions, targets, mask=None)`** — for graphs that already end in `.softmax()`; prefer the logits-based losses when you have logits.
+
+All three now share the same rules, closing several long-standing traps:
+
+- **One shape convention.** Every loss accepts `(N, V)` *and* `(B, T, V)` logits (auto-flattened internally); masks may be `(N,)` or `(B, T)`.
+- **One normalization convention.** Every loss divides by the number of *real* (unmasked) tokens — no more `/B` vs `/B·T` vs `/n_real` drift between variants, which silently changed effective learning rates when switching losses.
+- **Masks are optional everywhere**, defaulting to "all positions are real".
+- **Exact log-probabilities.** Loss values are computed from `log-softmax` directly instead of `log(clip(p))`, so near-zero probabilities are no longer floored at `log(1e-15)`.
+
+**Label smoothing**
+
+`sparse_softmax_cross_entropy_index(..., eps=0.1)` trains against `(1−eps)` on the true class and `eps/(V−1)` spread uniformly over the rest — the standard regularizer for translation models — at zero extra memory: both loss and gradient are computed straight from the label indices. `eps=0.0` (default) is exact plain cross-entropy.
+
+**Performance**
+
+- The hot path (`eps=0`, no mask — e.g. char-level GPT training) keeps the fused, **numexpr-accelerated** in-place backward from the previous `sparse_softmax_cross_entropy`.
+- Masked/smoothed paths fold the mask, upstream gradient, and normalization into a single per-row coefficient — one fewer `(N, V)` temporary in every backward.
+
+**Legacy module**
+
+The superseded losses were **moved, not deleted**: they live in `cogforge.legacy` under `TensorLegacy`, unchanged, for reference and for reproducing old experiments. New code should not import them — note that they retain their old (inconsistent) normalization conventions.
+
+---
+
+### Previous release — autograd correctness pass
+
+This version focused on **autograd correctness and debuggability**. All fixes below were validated with finite-difference gradient checks and end-to-end training tests.
 
 **Gradient correctness fixes**
 
@@ -82,6 +116,7 @@ The package is organized into four modules:
 | `cogforge.app` | The autograd engine (`Tensor`) and every building block — layers, optimizers, losses, normalization, attention, positional encodings. |
 | `cogforge.models` | Ready-to-use models: `GPTV1`, `GPT2`, `Seq2Seq`. |
 | `cogforge.utils` | Computation-graph tracing and Graphviz rendering (`trace_graph`, `draw_graph`). |
+| `cogforge.legacy` | `TensorLegacy` — the superseded cross-entropy variants, kept unchanged for reference and reproducing old experiments. Don't use in new code. |
 
 ```python
 from cogforge.app import Tensor, Linear, Adam, MultiHeadAttention   # building blocks
@@ -219,20 +254,23 @@ Gradients **accumulate** into `.grad`. Always zero them between optimization ste
 
 ### Losses
 
-All losses are **classmethods** on `Tensor` and return a scalar loss tensor you call `.backwards()` on. Mind the distinction between losses that take **probabilities** and losses that take **raw logits** — this is the most common mistake.
+All losses are **classmethods** on `Tensor` and return a scalar loss tensor you call `.backwards()` on. There are exactly **three**, and they share one convention:
 
-| Loss | Input expectation | Use when |
-| --- | --- | --- |
-| `Tensor.softmax_cross_entropy(scores, targets)` | `scores` are **raw logits**, `targets` one-hot. Softmax fused inside (stable). Works for 2-D `(B,V)` and 3-D `(B,T,V)`. | Standard classification / LM. **Recommended.** |
-| `Tensor.sparse_softmax_cross_entropy(scores, target_ids)` | `scores` raw logits `(B,T,V)`, `target_ids` integers `(B,T)`. | Language modeling — skips building one-hot targets. |
-| `Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask)` | Raw logits + integer labels + per-token mask (1 = real, 0 = pad). | Padded LM / seq2seq batches. **Recommended for Seq2Seq training.** |
-| `Tensor.softmax_cross_entropy_masked(scores, targets, mask)` | Logits + one-hot targets + per-row mask. | Padded batches, fused softmax. |
-| `Tensor.cross_entropy_loss(predictions, targets)` | `predictions` are **probabilities** (call `.softmax()` first), `targets` one-hot. | You already have a softmax in your graph. |
-| `Tensor.cross_entropy_loss_masked(predictions, targets, mask)` | Probabilities + per-row mask. | Padded batches without fused softmax. |
+- **Shapes:** logits/probs may be `(N, V)` or `(B, T, V)` — 3-D input is auto-flattened. Masks may be `(N,)` or `(B, T)`.
+- **Masks:** optional everywhere (`mask=None` treats all positions as real); `1` = real token, `0` = padding. Padded positions contribute zero loss *and* exactly zero gradient.
+- **Normalization:** always by the number of real tokens (all positions when unmasked).
 
-> ℹ️ The `softmax_*` and `sparse_softmax_*` variants apply softmax internally — feed them **raw logits**. `cross_entropy_loss*` is the opposite — it expects probabilities.
+Pick by what your **target** looks like and what your **input** is:
+
+| Loss | Input | Target | Use when |
+| --- | --- | --- | --- |
+| `Tensor.sparse_softmax_cross_entropy_index(scores, labels, mask=None, eps=0.0)` | **raw logits** | integer class ids | **The workhorse.** Classification, LM, seq2seq. `eps>0` enables label smoothing: `1−eps` on the true class, `eps/(V−1)` on the rest — computed straight from indices, no one-hot built. `eps=0.1` is the standard for translation. |
+| `Tensor.softmax_cross_entropy(scores, targets, mask=None)` | **raw logits** | full distribution `(…, V)`, rows sum to 1 | Targets that aren't expressible as an index: knowledge distillation against a teacher's output, mixup, annotator vote distributions. |
+| `Tensor.cross_entropy_from_probs(predictions, targets, mask=None)` | **probabilities** (e.g. output of `.softmax()`) | one-hot or distribution | Your graph already ends in a softmax. Prefer the logits-based losses when you have logits — their gradients are exact and bounded, this one saturates via clipping as probs → 0. |
+
+> ℹ️ The first two apply softmax internally — feed them **raw logits**. `cross_entropy_from_probs` is the opposite — it expects probabilities. Mixing these up silently trains the wrong thing.
 >
-> `sparse_softmax_cross_entropy_legacy` and `softmax_cross_entropy_old` are kept for reference; prefer the current versions.
+> The previous generation of losses (`softmax_cross_entropy_masked`, `sparse_softmax_cross_entropy`, `cross_entropy_loss`, `cross_entropy_loss_masked`, and the `_old`/`_legacy` variants) now lives in `cogforge.legacy` under `TensorLegacy`, unchanged. They keep their old, mutually inconsistent normalizations (`/B` vs `/B·T` vs `/n_real`) — use them only to reproduce old experiments, and don't mix them with the current losses in one training run.
 
 ---
 
@@ -436,7 +474,7 @@ def get_batch(bs=32):
 for step in range(2000):
     x, y   = get_batch()
     logits = model(x)                                  # (B, T, vocab)
-    loss   = Tensor.sparse_softmax_cross_entropy(logits, y)
+    loss   = Tensor.sparse_softmax_cross_entropy_index(logits, y)
 
     opt.zero_grad()
     loss.backwards()
@@ -489,8 +527,9 @@ for step in range(num_steps):
     dec_in, labels = tgt[:, :-1], tgt[:, 1:]
     logits = model(src, dec_in)            # (B, T_dec, vocab)
 
-    mask = (labels != PAD).astype(np.float32)
-    loss = Tensor.sparse_softmax_cross_entropy_index(logits, labels, mask)
+    mask = (labels != PAD).astype(np.float32)     # (B, T_dec): 1 = real, 0 = pad
+    loss = Tensor.sparse_softmax_cross_entropy_index(logits, labels,
+                                                     mask=mask, eps=0.1)  # label smoothing
 
     opt.zero_grad()
     loss.backwards()
@@ -535,7 +574,9 @@ Two practical notes: call `draw_graph` **before** `backwards()`, since the backw
 ## Gotchas
 
 - **It's `backwards()`, not `backward()`.** The backward pass method has a trailing `s`.
-- **Logits vs. probabilities.** `softmax_cross_entropy` / `sparse_softmax_cross_entropy*` fuse the softmax internally — feed them **raw logits**. `cross_entropy_loss*` expects **probabilities**. Mixing these up silently trains the wrong thing.
+- **Logits vs. probabilities.** `softmax_cross_entropy` and `sparse_softmax_cross_entropy_index` fuse the softmax internally — feed them **raw logits**. `cross_entropy_from_probs` expects **probabilities**. Mixing these up silently trains the wrong thing.
+- **Label smoothing raises the loss floor.** With `eps > 0` the minimum achievable loss is no longer 0 (a perfect model still pays the smoothing term), so judge training by token accuracy or validation metrics, not by how close the raw loss gets to zero — and don't compare loss values across different `eps` settings.
+- **Legacy losses normalize differently.** `cogforge.legacy.TensorLegacy` keeps the old `/B` / `/B·T` conventions; the current losses always divide by real-token count. Swapping one for the other rescales gradients — retune the learning rate if you migrate an old script.
 - **Gradients accumulate.** Call `optimizer.zero_grad()` every step (or `p.grad[...] = 0`), or gradients pile up across iterations.
 - **Switch the backend before building the model.** `use_gpu(True)` after construction leaves your parameters stranded on the CPU while new activations land on the GPU.
 - **RoPE dim is per-head.** `RotatoryPositionalEncoding` takes `d_model // n_heads` (which must be even), not `d_model`. The models handle this internally — it only matters if you wire blocks up yourself.
@@ -552,13 +593,17 @@ Two practical notes: call `draw_graph` **before** `backwards()`, since the backw
 
 Shipped in this release:
 
-- ✅ **Autograd correctness pass** — all gradients verified against finite-difference checks; broadcast, indexing, sparse cross-entropy, and BatchNorm backward bugs fixed (see [What's new](#whats-new))
-- ✅ Computation-graph visualization via Graphviz (`cogforge.utils`), with per-tensor `op` labels
-- ✅ Per-sequence EOS handling in `Seq2Seq.generate` (finished rows freeze to `pad_id`)
-- ✅ Eager graph freeing in `backwards()` for lower peak memory
+- ✅ **Unified loss API** — three cross-entropy losses covering index targets, distribution targets, and probability inputs, with one shape and normalization convention (see [What's new](#whats-new))
+- ✅ **Label smoothing** (`eps`) in `sparse_softmax_cross_entropy_index`, computed from indices with zero extra memory
+- ✅ Optional padding masks and `(B, T, V)` auto-flattening in every loss
+- ✅ `cogforge.legacy` module (`TensorLegacy`) preserving the superseded losses for reproducibility
 
 Shipped previously:
 
+- ✅ **Autograd correctness pass** — all gradients verified against finite-difference checks; broadcast, indexing, sparse cross-entropy, and BatchNorm backward bugs fixed
+- ✅ Computation-graph visualization via Graphviz (`cogforge.utils`), with per-tensor `op` labels
+- ✅ Per-sequence EOS handling in `Seq2Seq.generate` (finished rows freeze to `pad_id`)
+- ✅ Eager graph freeing in `backwards()` for lower peak memory
 - ✅ RoPE (rotary position embeddings), usable in GPT and at every attention site of the Seq2Seq model
 - ✅ Full encoder–decoder transformer (`Seq2Seq`) with cross-attention and padding masks
 - ✅ Weight tying between embeddings and the output head
@@ -568,6 +613,7 @@ Shipped previously:
 
 Planned / under consideration:
 
+- Beam search decoding for `Seq2Seq` (length-normalized, per-sequence finished pool)
 - Lazy Gradient Allocation (LGA) for lower peak memory on long sequences
 - Cosine LR function as part of UTILS module
 - Weight decay / AdamW optimizer

{cogforge_engine-2.1.2 → cogforge_engine-2.1.3}/TODO

@@ -6,4 +6,6 @@ cosine LR function
 
 Batch acummalator normalization
 
-The 50% Memory Tax (Lazy Gradient Allocation)
+The 50% Memory Tax (Lazy Gradient Allocation)
+
+Beam search decoding with length penalty

{cogforge_engine-2.1.2 → cogforge_engine-2.1.3}/cogforge/app.py

@@ -279,25 +279,6 @@ class Tensor:
         out._backwards = _backward
         return out
     
-    #optimized for batches
-    @classmethod
-    def cross_entropy_loss(cls,predictions, targets):
-       
-        probabilities = backend.np.clip(predictions.data, 1e-15, 1 - 1e-15)
-        batch_size = predictions.shape[0]
-        
-        loss_data = - backend.np.sum(targets * backend.np.log(probabilities)) / batch_size
-        if backend.NO_GRAD: return cls(loss_data,op="cross_entropy_loss")
-        loss = Tensor(loss_data, children=(predictions,),op="cross_entropy_loss")
-        
-        def _backward():
-            
-            # predictions.grad += (predictions.data - targets) / batch_size * loss.grad
-            predictions.grad += (-(targets / probabilities)) / batch_size * loss.grad
-
-            
-        loss._backwards = _backward
-        return loss
     
     #LEGACY
     def backwards_recursive(self):
@@ -396,186 +377,139 @@ class Tensor:
     
     #TODO
     @classmethod
-    def cross_entropy_loss_masked(cls, predictions, targets, mask):
+    def cross_entropy_from_probs(cls, predictions, targets, mask=None):
         """
-        predictions: (B, V) softmax probs
-        targets:     (B, V) one-hot
-        mask:        (B,) 1.0 for real positions, 0.0 for <PAD>
+        predictions: Tensor (N, V) or (B, T, V) — probabilities (rows sum to 1),
+                    e.g. output of .softmax(). Prefer the logits-based losses
+                    (sparse_softmax_cross_entropy_index / softmax_cross_entropy)
+                    when you have logits: their gradients are exact and bounded,
+                    this one saturates via the clip when probs -> 0.
+        targets:     array, same shape — one-hot or arbitrary distribution
+        mask:        optional (N,) or (B, T); 1 = real, 0 = pad
         """
-        probs = backend.np.clip(predictions.data, 1e-15, 1 - 1e-15)
-        mask = mask.reshape(-1, 1)                          # (B, 1)
+        orig_shape = predictions.data.shape
+        V = orig_shape[-1]
+        N = int(numpy.prod(orig_shape[:-1]))
+
+        probs = backend.np.clip(predictions.data.reshape(N, V), 1e-15, 1 - 1e-15)
+        q = backend.np.asarray(targets).reshape(N, V)
+
+        if mask is None:
+            mask = backend.np.ones(N, dtype=probs.dtype)
+        else:
+            mask = backend.np.asarray(mask).reshape(-1).astype(probs.dtype)
         n_real = mask.sum()
         n_real = n_real if n_real > 0 else 1.0
 
-        # only real rows contribute to the loss value
-        loss_data = -backend.np.sum(mask * targets * backend.np.log(probs)) / n_real
-        
-        if backend.NO_GRAD: return cls(loss_data)
-        loss = cls(loss_data, children=(predictions,))
+        loss_data = -backend.np.sum(mask[:, None] * q * backend.np.log(probs)) / n_real
 
-        def _backward():
-            # grad = (predictions.data - targets) / n_real
-            grad = -(targets / probs) / n_real
-            grad = grad * mask                              # <-- padded rows get exactly zero gradient
-            predictions.grad += grad * loss.grad
-
-        loss._backwards = _backward
-        return loss
+        if backend.NO_GRAD:
+            return cls(loss_data, op="cross_entropy_from_probs")
+        loss = cls(loss_data, children=(predictions,), op="cross_entropy_from_probs")
 
-    @classmethod
-    def sparse_softmax_cross_entropy_legacy(cls, scores, target_ids):
-        """ scores: (B, T, V) logits ; target_ids: (B, T) int """
-        z = scores.data - scores.data.max(axis=-1, keepdims=True)
-        e = backend.np.exp(z)
-        p = e / e.sum(axis=-1, keepdims=True)
-        N = int(numpy.prod(scores.shape[:-1]))
-        flat, idx, rows = p.reshape(N, -1), target_ids.reshape(N), backend.np.arange(N)
-        # loss = cls(-backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N, children=(scores,))
-        
-        loss_data = -backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N
-        
-        if backend.NO_GRAD: return cls(loss_data)
-        
-        loss = cls(loss_data, children=(scores,))
-        
         def _backward():
-            # g = p.reshape(N, -1).copy()
-            g = p.reshape(N, -1)
-            g[rows, idx] -= 1.0                      # (softmax - onehot)
-            scores.grad += (g.reshape(scores.shape) / N) * loss.grad
-        loss._backwards = _backward
-        return loss
-    
-    @classmethod
-    def sparse_softmax_cross_entropy(cls, scores, target_ids):
-        p = _stable_softmax(scores.data)
-        N = int(numpy.prod(scores.shape[:-1]))
-        flat, idx, rows = p.reshape(N, -1), target_ids.reshape(N), backend.np.arange(N)
-        # loss = cls(-backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N, children=(scores,))
-        
-        loss_data = -backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N
-        
-        if backend.NO_GRAD: return cls(loss_data, op="sparse_softmax_crossentropy")
-        
-        loss = cls(loss_data, children=(scores,), op="sparse_softmax_crossentropy")
-        
-        def _backward():
-            # g = p.reshape(N, -1).copy() #added copy
-            # g[rows, idx] -= 1.0
-            # coef = numpy.float32(float(loss.grad) / N)      
-            # _iadd_scaled(scores.grad, g.reshape(scores.shape), coef)
-            
-            coef = numpy.float32(float(loss.grad) / N)
-            _iadd_scaled(scores.grad, p.reshape(scores.shape), coef) 
-            flat_grad = scores.grad.reshape(N, -1) 
-            flat_grad[rows, idx] -= coef
+            row_coef = (mask * loss.grad) / n_real           
+            grad = -(q / probs) * row_coef[:, None]
+            predictions.grad += grad.reshape(orig_shape)
         loss._backwards = _backward
         return loss
+
     
     @classmethod
-    def sparse_softmax_cross_entropy_index(cls, scores, labels, mask):
-        """
-        scores: Tensor (N, V) logits
-        labels: (N,) int   correct class id per row
-        mask:   (N,) {0,1}  1 = real token, 0 = padding
-        Same loss/gradient as the one-hot masked version, no (N, V) target built.
-        """
-        z = scores.data
-        z = z - z.max(axis=1, keepdims=True)
+    def sparse_softmax_cross_entropy_index(cls, scores, labels, mask=None, eps=0.0):
+        orig_shape = scores.data.shape
+        V = orig_shape[-1]
+        N = int(numpy.prod(orig_shape[:-1]))                 # (B,T,V) and (N,V) both fine
+
+        z = scores.data.reshape(N, V)
+        z = z - z.max(axis=-1, keepdims=True)
         e = backend.np.exp(z)
-        p = e / e.sum(axis=1, keepdims=True)              # (N, V)
+        denom = e.sum(axis=-1, keepdims=True)
+        p = e / denom
+        logp = z - backend.np.log(denom)
 
-        labels = backend.np.asarray(labels).reshape(-1)   # (N,)
-        mask   = backend.np.asarray(mask).reshape(-1)     # (N,)
-        N = p.shape[0]
+        labels = backend.np.asarray(labels).reshape(-1)
         rows = backend.np.arange(N)
 
+        if mask is None:
+            mask = backend.np.ones(N, dtype=p.dtype)
+        else:
+            mask = backend.np.asarray(mask).reshape(-1).astype(p.dtype)
         n_real = mask.sum()
         n_real = n_real if n_real > 0 else 1.0
 
-        logp_correct = backend.np.log(backend.np.clip(p[rows, labels], 1e-15, 1.0))
-        loss_data = -backend.np.sum(mask * logp_correct) / n_real
+        logp_correct = logp[rows, labels]
+        if eps > 0.0:
+            off = eps / (V - 1)
+            per_tok = -((1.0 - eps) * logp_correct + off * (logp.sum(axis=-1) - logp_correct))
+        else:
+            per_tok = -logp_correct
+        loss_data = backend.np.sum(mask * per_tok) / n_real
 
         if backend.NO_GRAD:
-            return cls(loss_data,op="sparse_softmax_crossentropy")
+            return cls(loss_data, op="sparse_softmax_crossentropy")
+        loss = cls(loss_data, children=(scores,), op="sparse_softmax_crossentropy")
 
-        loss = cls(loss_data, children=(scores,),op="sparse_softmax_crossentropy")
         def _backward():
-            # grad = p.copy()                    
-            # grad[rows, labels] -= 1.0          
-            # grad *= (mask / n_real)[:, None]   
-            # scores.grad += grad * loss.grad
-            
-            row_coef = (mask * loss.grad) / n_real
-            row_coef_expanded = row_coef[:, None] 
-            scores.grad += p * row_coef_expanded
-            scores.grad[rows, labels] -= row_coef
-        loss._backwards = _backward
-        return loss
-    
-    #Legacy
-    @classmethod
-    def softmax_cross_entropy_old(cls, scores, targets):
-        z = scores.data
-        z = z - z.max(axis=1, keepdims=True)          # stability
-        e = backend.np.exp(z)
-        p = e / e.sum(axis=1, keepdims=True)          # softmax, computed privately
-        B = scores.shape[0]
-
-        # loss = cls(-backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / B, children=(scores,))
+            no_mask = mask.min() == 1.0                      
+            flat_grad = scores.grad.reshape(N, V)
 
-        loss_data = -backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / B
-        
-        if backend.NO_GRAD: return cls(loss_data)
-        
-        loss = cls(loss_data, children=(scores,))
-        
-        def _backward():
-            scores.grad += (p - targets) / B * loss.grad
+            if eps == 0.0 and no_mask:
+                coef = numpy.float32(float(loss.grad) / n_real)
+                _iadd_scaled(flat_grad, p, coef)
+                flat_grad[rows, labels] -= coef
+            else:
+                row_coef = (mask * loss.grad) / n_real       
+                base = p - eps / (V - 1) if eps > 0.0 else p
+                flat_grad += base * row_coef[:, None]
+                spike = (1.0 - eps) - (eps / (V - 1)) if eps > 0.0 else 1.0
+                flat_grad[rows, labels] -= spike * row_coef
         loss._backwards = _backward
         return loss
     
+   
     @classmethod
-    def softmax_cross_entropy(cls, scores, targets):
-        z = scores.data
-        z = z - z.max(axis=-1, keepdims=True)          
-        e = backend.np.exp(z)
-        p = e / e.sum(axis=-1, keepdims=True)
-        N = numpy.prod(scores.shape[:-1])                  # B for 2D, B*T for 3D
-        # loss = cls(-backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / N, children=(scores,))
-        loss_data = -backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / N        
-        if backend.NO_GRAD: return cls(loss_data)        
-        loss = cls(loss_data, children=(scores,))
-        def _backward():
-            scores.grad += (p - targets) / N * loss.grad
-        loss._backwards = _backward
-        return loss
-
+    def softmax_cross_entropy(cls, scores, targets, mask=None):
+        """
+        scores:  Tensor (N, V) or (B, T, V) logits
+        targets: array, same shape — arbitrary distribution per row (rows sum to 1)
+        mask:    optional (N,) or (B, T); 1 = real, 0 = pad
+        """
+        orig_shape = scores.data.shape
+        V = orig_shape[-1]
+        N = int(numpy.prod(orig_shape[:-1]))                 
 
-    @classmethod
-    def softmax_cross_entropy_masked(cls, scores, targets, mask):
-        z = scores.data
-        z = z - z.max(axis=1, keepdims=True)
+        z = scores.data.reshape(N, V)
+        z = z - z.max(axis=-1, keepdims=True)
         e = backend.np.exp(z)
-        p = e / e.sum(axis=1, keepdims=True)
+        denom = e.sum(axis=-1, keepdims=True)
+        p = e / denom
+        logp = z - backend.np.log(denom)                     
 
-        mask = mask.reshape(-1, 1)                     
+        q = backend.np.asarray(targets).reshape(N, V)
+
+        if mask is None:
+            mask = backend.np.ones(N, dtype=p.dtype)
+        else:
+            mask = backend.np.asarray(mask).reshape(-1).astype(p.dtype)
         n_real = mask.sum()
         n_real = n_real if n_real > 0 else 1.0
 
-        # loss = cls(-backend.np.sum(mask * targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / n_real, children=(scores,))
-        loss_data = -backend.np.sum(mask * targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / n_real
-        
-        if backend.NO_GRAD: return cls(loss_data)
-        
-        loss = cls(loss_data, children=(scores,))
-        
+        loss_data = -backend.np.sum(mask[:, None] * q * logp) / n_real
+
+        if backend.NO_GRAD:
+            return cls(loss_data, op="soft_cross_entropy")
+        loss = cls(loss_data, children=(scores,), op="soft_cross_entropy")
+
         def _backward():
-            grad = (p - targets) / n_real
-            grad = grad * mask                         
-            scores.grad += grad * loss.grad
+            row_coef = (mask * loss.grad) / n_real          
+            grad = (p - q) * row_coef[:, None]
+            scores.grad += grad.reshape(orig_shape)
+            
         loss._backwards = _backward
         return loss
+
+
     
     def transpose(self,axes):
         if backend.NO_GRAD: return Tensor(backend.np.transpose(self.data,axes=axes),op=f"transpose[{axes}]")

cogforge_engine-2.1.3/cogforge/legacy.py

@@ -0,0 +1,179 @@
+from . import backend
+import numpy
+from .app import _iadd_scaled,_stable_softmax,Tensor
+
+class TensorLegacy:
+     #Legacy DONT INCLUDE IN DOCS
+    @classmethod
+    def sparse_softmax_cross_entropy_index_legacy(cls, scores, labels, mask):
+        """
+        scores: Tensor (N, V) logits
+        labels: (N,) int   correct class id per row
+        mask:   (N,) {0,1}  1 = real token, 0 = padding
+        Same loss/gradient as the one-hot masked version, no (N, V) target built.
+        """
+        z = scores.data
+        z = z - z.max(axis=1, keepdims=True)
+        e = backend.np.exp(z)
+        p = e / e.sum(axis=1, keepdims=True)              # (N, V)
+
+        labels = backend.np.asarray(labels).reshape(-1)   # (N,)
+        mask   = backend.np.asarray(mask).reshape(-1)     # (N,)
+        N = p.shape[0]
+        rows = backend.np.arange(N)
+
+        n_real = mask.sum()
+        n_real = n_real if n_real > 0 else 1.0
+
+        logp_correct = backend.np.log(backend.np.clip(p[rows, labels], 1e-15, 1.0))
+        loss_data = -backend.np.sum(mask * logp_correct) / n_real
+
+        if backend.NO_GRAD:
+            return cls(loss_data,op="sparse_softmax_crossentropy")
+
+        loss = cls(loss_data, children=(scores,),op="sparse_softmax_crossentropy")
+        def _backward():
+            # grad = p.copy()                    
+            # grad[rows, labels] -= 1.0          
+            # grad *= (mask / n_real)[:, None]   
+            # scores.grad += grad * loss.grad
+            
+            row_coef = (mask * loss.grad) / n_real
+            row_coef_expanded = row_coef[:, None] 
+            scores.grad += p * row_coef_expanded
+            scores.grad[rows, labels] -= row_coef
+        loss._backwards = _backward
+        return loss
+    #Legacy
+    @classmethod
+    def softmax_cross_entropy_old(cls, scores, targets):
+        z = scores.data
+        z = z - z.max(axis=1, keepdims=True)          # stability
+        e = backend.np.exp(z)
+        p = e / e.sum(axis=1, keepdims=True)          # softmax, computed privately
+        B = scores.shape[0]
+
+        # loss = cls(-backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / B, children=(scores,))
+
+        loss_data = -backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / B
+        
+        if backend.NO_GRAD: return cls(loss_data)
+        
+        loss = cls(loss_data, children=(scores,))
+        
+        def _backward():
+            scores.grad += (p - targets) / B * loss.grad
+        loss._backwards = _backward
+        return loss
+    
+    @classmethod
+    def softmax_cross_entropy_masked(cls, scores, targets, mask):
+        z = scores.data
+        z = z - z.max(axis=1, keepdims=True)
+        e = backend.np.exp(z)
+        p = e / e.sum(axis=1, keepdims=True)
+
+        mask = mask.reshape(-1, 1)                     
+        n_real = mask.sum()
+        n_real = n_real if n_real > 0 else 1.0
+
+        # loss = cls(-backend.np.sum(mask * targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / n_real, children=(scores,))
+        loss_data = -backend.np.sum(mask * targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / n_real
+        
+        if backend.NO_GRAD: return cls(loss_data)
+        
+        loss = cls(loss_data, children=(scores,))
+        
+        def _backward():
+            grad = (p - targets) / n_real
+            grad = grad * mask                         
+            scores.grad += grad * loss.grad
+        loss._backwards = _backward
+        return loss
+    
+    #Older Version. Newer index function should be used for eps improvement. Dont use even without padding, write clearly in docs
+    @classmethod
+    def sparse_softmax_cross_entropy(cls, scores, target_ids):
+        p = _stable_softmax(scores.data)
+        N = int(numpy.prod(scores.shape[:-1]))
+        flat, idx, rows = p.reshape(N, -1), target_ids.reshape(N), backend.np.arange(N)
+        # loss = cls(-backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N, children=(scores,))
+        
+        loss_data = -backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N
+        
+        if backend.NO_GRAD: return cls(loss_data, op="sparse_softmax_crossentropy")
+        
+        loss = cls(loss_data, children=(scores,), op="sparse_softmax_crossentropy")
+        
+        def _backward():
+            # g = p.reshape(N, -1).copy() #added copy
+            # g[rows, idx] -= 1.0
+            # coef = numpy.float32(float(loss.grad) / N)      
+            # _iadd_scaled(scores.grad, g.reshape(scores.shape), coef)
+            
+            coef = numpy.float32(float(loss.grad) / N)
+            _iadd_scaled(scores.grad, p.reshape(scores.shape), coef) 
+            flat_grad = scores.grad.reshape(N, -1) 
+            flat_grad[rows, idx] -= coef
+        loss._backwards = _backward
+        return loss
+    
+    @classmethod
+    def sparse_softmax_cross_entropy_legacy(cls, scores, target_ids):
+        """ scores: (B, T, V) logits ; target_ids: (B, T) int """
+        z = scores.data - scores.data.max(axis=-1, keepdims=True)
+        e = backend.np.exp(z)
+        p = e / e.sum(axis=-1, keepdims=True)
+        N = int(numpy.prod(scores.shape[:-1]))
+        flat, idx, rows = p.reshape(N, -1), target_ids.reshape(N), backend.np.arange(N)
+        # loss = cls(-backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N, children=(scores,))
+        
+        loss_data = -backend.np.log(backend.np.clip(flat[rows, idx], 1e-15, 1.0)).sum() / N
+        
+        if backend.NO_GRAD: return cls(loss_data)
+        
+        loss = cls(loss_data, children=(scores,))
+        
+        def _backward():
+            # g = p.reshape(N, -1).copy()
+            g = p.reshape(N, -1)
+            g[rows, idx] -= 1.0                      # (softmax - onehot)
+            scores.grad += (g.reshape(scores.shape) / N) * loss.grad
+        loss._backwards = _backward
+        return loss
+    
+    @classmethod
+    def softmax_cross_entropy(cls, scores, targets):
+        z = scores.data
+        z = z - z.max(axis=-1, keepdims=True)          
+        e = backend.np.exp(z)
+        p = e / e.sum(axis=-1, keepdims=True)
+        N = numpy.prod(scores.shape[:-1])                  # B for 2D, B*T for 3D
+        # loss = cls(-backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / N, children=(scores,))
+        loss_data = -backend.np.sum(targets * backend.np.log(backend.np.clip(p, 1e-15, 1.0))) / N        
+        if backend.NO_GRAD: return cls(loss_data)        
+        loss = cls(loss_data, children=(scores,))
+        def _backward():
+            scores.grad += (p - targets) / N * loss.grad
+        loss._backwards = _backward
+        return loss
+    
+    #optimized for batches
+    @classmethod
+    def cross_entropy_loss(cls,predictions, targets):
+       
+        probabilities = backend.np.clip(predictions.data, 1e-15, 1 - 1e-15)
+        batch_size = predictions.shape[0]
+        
+        loss_data = - backend.np.sum(targets * backend.np.log(probabilities)) / batch_size
+        if backend.NO_GRAD: return cls(loss_data,op="cross_entropy_loss")
+        loss = Tensor(loss_data, children=(predictions,),op="cross_entropy_loss")
+        
+        def _backward():
+            
+            # predictions.grad += (predictions.data - targets) / batch_size * loss.grad
+            predictions.grad += (-(targets / probabilities)) / batch_size * loss.grad
+
+            
+        loss._backwards = _backward
+        return loss

{cogforge_engine-2.1.2 → cogforge_engine-2.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "cogforge-engine" 
-version = "2.1.2"
+version = "2.1.3"
 authors = [
   { name="Avik Majumder", email="avikmjd2@gmail.com" },
 ]