dirichlet-regularization 1.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 mcarbonell / Topographic Spectral Research Group
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,292 @@
1
+ Metadata-Version: 2.4
2
+ Name: dirichlet-regularization
3
+ Version: 1.0.0
4
+ Summary: 2D Dirichlet spatial regularization for neural network weights: inducing topographic smoothness for 2D-DCT spectral compression and Base-3 trit quantization
5
+ Author-email: Mario Raúl Carbonell Martínez <marioraulcarbonell@gmail.com>
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/mcarbonell/dirichlet-regularization
8
+ Project-URL: Repository, https://github.com/mcarbonell/dirichlet-regularization
9
+ Keywords: deep-learning,pytorch,regularization,dirichlet-energy,dct,quantization,model-compression
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Science/Research
12
+ Classifier: Programming Language :: Python :: 3
13
+ Classifier: Programming Language :: Python :: 3.9
14
+ Classifier: Programming Language :: Python :: 3.10
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: C
18
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
19
+ Classifier: Operating System :: OS Independent
20
+ Requires-Python: >=3.9
21
+ Description-Content-Type: text/markdown
22
+ License-File: LICENSE
23
+ Requires-Dist: torch>=2.0.0
24
+ Requires-Dist: numpy>=1.20.0
25
+ Provides-Extra: examples
26
+ Requires-Dist: torchvision>=0.15.0; extra == "examples"
27
+ Requires-Dist: matplotlib>=3.5.0; extra == "examples"
28
+ Provides-Extra: benchmarks
29
+ Requires-Dist: modal>=0.50.0; extra == "benchmarks"
30
+ Requires-Dist: datasets>=2.0.0; extra == "benchmarks"
31
+ Requires-Dist: tokenizers>=0.13.0; extra == "benchmarks"
32
+ Requires-Dist: transformers>=4.28.0; extra == "benchmarks"
33
+ Requires-Dist: matplotlib>=3.5.0; extra == "benchmarks"
34
+ Requires-Dist: tqdm>=4.65.0; extra == "benchmarks"
35
+ Provides-Extra: dev
36
+ Requires-Dist: pytest>=7.0.0; extra == "dev"
37
+ Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
38
+ Requires-Dist: ruff>=0.1.0; extra == "dev"
39
+ Dynamic: license-file
40
+
41
+ <div align="center">
42
+
43
+ # Dirichlet Regularization (`dreg`)
44
+ ### Spatial Smoothness in Neural Weight Manifolds for 2D-DCT Spectral Compression
45
+
46
+ [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
47
+ [![PyTorch 2.0+](https://img.shields.io/badge/pytorch-2.0+-red.svg)](https://pytorch.org/)
48
+ [![CI](https://github.com/mcarbonell/dirichlet-regularization/actions/workflows/ci.yml/badge.svg)](https://github.com/mcarbonell/dirichlet-regularization/actions/workflows/ci.yml)
49
+ [![Tests](https://img.shields.io/badge/tests-99_passed-brightgreen.svg)](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
50
+ [![Coverage](https://img.shields.io/badge/coverage-96%25-brightgreen.svg)](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
51
+ [![Code Style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
52
+ [![Preprint (PDF)](https://img.shields.io/badge/Preprint-PDF-purple.svg)](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf)
53
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE)
54
+
55
+ </div>
56
+
57
+ ---
58
+
59
+ ## The Core Idea
60
+
61
+ Standard neural networks treat weight matrices as unstructured, permutation-invariant point clouds. After training with isotropic $L_2$ weight decay, their 2D spatial frequency spectrum resembles **flat white noise** — variance is spread uniformly across all frequencies, making frequency-domain compression ineffective.
62
+
63
+ **Dirichlet Regularization (`dreg`)** introduces a simple inductive bias during training: *neighboring entries on a 2D weight lattice should vary smoothly*, analogous to topographic organization in biological cortex.
64
+
65
+ Penalizing discrete 2D spatial differences breaks permutation symmetry and **concentrates spectral energy in the low-frequency harmonics of the orthonormal 2D-DCT** — up to **~72%** of Frobenius energy below $\rho \le 0.40$ on square Attention projections (`256×256`) and **52.8%** on MNIST MLPs — enabling hierarchical Base-3 Trit spectral quantization (`.tritq`) across a configurable `0.43–3.77 bpp` rate–distortion frontier.
66
+
67
+ <p align="center">
68
+ <img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig1_weight_heatmaps.png" alt="Spatial Weight Structures: Standard AdamW vs Dirichlet Regularization" width="90%">
69
+ </p>
70
+
71
+ > **Figure 1: Illustrative Spatial Weight Structures (MNIST MLP, 64×64 top-left patches).** 3-epoch MNIST MLP (`fc1` top, `fc2` bottom, unnormalized $\epsilon = 2 \times 10^{-3}$). Left: Standard AdamW yields high-frequency unstructured weights (edge-normalized $E_D = 0.0016$ on `fc1`). Right: 2D Dirichlet regularization induces smooth spatial manifolds ($E_D = 0.0001$, $-94\%$ roughness). On full TinyStories-10M Transformers (Table 2 of the preprint), mean $E_D$ drops by **$-72.1\%$** ($0.0123 \to 0.0034$ `numel`; $0.0062 \to 0.0017$ `edges`).
72
+
73
+ ---
74
+
75
+ ## The Mathematics
76
+
77
+ Standard $L_2$ weight decay penalizes **magnitude** and is strictly invariant to row/column permutations:
78
+
79
+ $$\mathcal{L}_{L_2}(W) = \frac{\gamma}{2} \|W\|_F^2$$
80
+
81
+ **Dirichlet Regularization** penalizes **2D discrete spatial differences** across the $M \times N$ weight lattice via the separable Neumann graph Laplacians $L_M \in \mathbb{R}^{M \times M}$ and $L_N \in \mathbb{R}^{N \times N}$:
82
+
83
+ $$\mathcal{E}_D^{\text{var}}(W) = \frac{1}{2MN}\!\left(\sum_{i=1}^{M-1}\sum_{j=1}^N (W_{i+1,j}-W_{i,j})^2 + \sum_{i=1}^M\sum_{j=1}^{N-1} (W_{i,j+1}-W_{i,j})^2\right) = \frac{1}{2MN}\Big(\operatorname{Tr}(W^\top L_M W) + \operatorname{Tr}(W L_N W^\top)\Big)$$
84
+
85
+ Because the eigenvectors of the 1D Neumann path-graph Laplacian are exactly the orthonormal **Type-II DCT** basis vectors, $\mathcal{E}_D$ diagonalizes in the 2D-DCT domain $S = D_M W D_N^\top$ (`numel` convention $= 2\,\mathcal{E}_D^{\text{var}}$):
86
+
87
+ $$\mathcal{E}_D^{\text{numel}}(W) = \frac{1}{MN} \sum_{k=0}^{M-1}\sum_{l=0}^{N-1} 4\!\left[\sin^2\!\left(\frac{\pi k}{2M}\right) + \sin^2\!\left(\frac{\pi l}{2N}\right)\right] S_{k,l}^2$$
88
+
89
+ High-frequency modes $(k, l)$ are penalized quadratically, concentrating weight energy into the lowest 2D-DCT harmonics.
90
+
91
+ > **Normalization note:** `dreg` provides `normalization="numel"` ($\frac{1}{MN}\sum \Delta^2 = 2\,\mathcal{E}_D^{\text{var}}$, default), `normalization="edges"` ($\approx \mathcal{E}_D^{\text{var}}$), and scale-invariant `normalization="rayleigh"` ($\frac{\sum \Delta^2}{2\|W\|_F^2} \in [0, 4]$, immune to $L_2$ magnitude shrinkage), plus `layer_weighting="param_weighted"` to balance gradient strength equally across square Attention and wide FFN layers. See [dreg/topology.py](https://github.com/mcarbonell/dirichlet-regularization/blob/main/dreg/topology.py) and §3.1 of the preprint.
92
+
93
+ <p align="center">
94
+ <img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig2_dct_energy_spectra.png" alt="2D-DCT Spectral Energy Compaction" width="90%">
95
+ </p>
96
+
97
+ > **Figure 2: 2D-DCT Cumulative Spectral Energy (TinyStories-10M, Layer 3).** Cumulative Frobenius energy vs. normalized radius $\rho \in [0, \sqrt{2}]$. Standard AdamW stays flat ($\sim 13\text{--}14\%$ below $\rho = 0.40$). Dirichlet regularization concentrates **~72%** of energy below $\rho = 0.40$ in Attention $W_q$ (`256×256`), and **~25%** in the $4\times$ larger FFN gate projection (`256×1024`) due to $1/(MN)$ per-weight gradient scaling.
98
+
99
+ ---
100
+
101
+ ## Quick Start
102
+
103
+ ### Installation
104
+
105
+ ```bash
106
+ pip install dirichlet-regularization
107
+ ```
108
+
109
+ Or from source:
110
+ ```bash
111
+ git clone https://github.com/mcarbonell/dirichlet-regularization.git
112
+ cd dirichlet-regularization
113
+ pip install -e .
114
+ ```
115
+
116
+ Optional extras:
117
+ ```bash
118
+ pip install -e ".[examples]" # torchvision + matplotlib for examples
119
+ pip install -e ".[dev]" # pytest, pytest-cov, ruff
120
+ ```
121
+
122
+ ### Apply to Any PyTorch Model (2 lines of code)
123
+
124
+ #### Option A: Universal loss wrapper (zero model modifications)
125
+ ```python
126
+ from dreg import DirichletLoss
127
+
128
+ # Automatically skips nn.Embedding, lm_head, frozen weights, and deduplicates tied weights.
129
+ # Use normalization="rayleigh" + layer_weighting="param_weighted" for scale-invariant,
130
+ # FFN-balanced smoothing (recommended: weight_decay=0.25–0.50), or classic normalization="numel"
131
+ # with weight_decay=15.0–30.0.
132
+ topo_reg = DirichletLoss(
133
+ weight_decay=0.50,
134
+ normalization="rayleigh",
135
+ layer_weighting="param_weighted",
136
+ )
137
+
138
+ # Inside your training loop:
139
+ task_loss = criterion(model(inputs), targets)
140
+ total_loss = task_loss + topo_reg(model)
141
+ total_loss.backward()
142
+ ```
143
+
144
+ #### Option B: Drop-in replacement for `nn.Linear`
145
+ ```python
146
+ import torch.nn as nn
147
+ from dreg import TopographicLinear
148
+
149
+ class MyModel(nn.Module):
150
+ def __init__(self):
151
+ super().__init__()
152
+ self.fc1 = TopographicLinear(512, 2048)
153
+ self.fc2 = TopographicLinear(2048, 10)
154
+ self.act = nn.GELU()
155
+
156
+ def forward(self, x):
157
+ return self.fc2(self.act(self.fc1(x)))
158
+
159
+ def dirichlet_loss(self):
160
+ return 0.5 * (self.fc1.dirichlet_energy() + self.fc2.dirichlet_energy())
161
+
162
+ model = MyModel()
163
+ loss = criterion(model(x), y) + 15.0 * model.dirichlet_loss()
164
+ loss.backward()
165
+ ```
166
+
167
+ > **Calibration tip:** Because `dirichlet_energy_2d` is normalized by $MN$, raw values are $\sim 10^{-3}\text{--}10^{-2}$. Use $\lambda_{\text{topo}} \in [5.0, 30.0]$ (rather than `0.01`, which is washed out by AdamW).
168
+
169
+ ---
170
+
171
+ ## Empirical Highlights
172
+
173
+ ### 1. TinyStories 10.56M Transformer (81.92M tokens, A10G GPU)
174
+
175
+ Evaluated on 25 validation batches (204,800 tokens; full results in [results/pareto_curve_tinystories_results.json](https://github.com/mcarbonell/dirichlet-regularization/blob/main/results/pareto_curve_tinystories_results.json) and Preprint Table 5):
176
+
177
+ | Quantization Regime | Radii $(r_0, r_1, r_2)$ | Linear Rate | Compression | Std PPL | Topo PPL ($\lambda=30$) | Dirichlet Gain |
178
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
179
+ | Dense FP32 (Reference) | — | 32.000 bpp | 1.0× | **6.11** | 6.46 | $-0.35$ PPL |
180
+ | Conservative | `(0.45, 0.90, 1.41)` | 3.774 bpp | 8.5× | 8.87 | **8.22** | **+0.65 PPL** |
181
+ | High-Fidelity | `(0.30, 0.70, 1.30)` | 2.794 bpp | 11.5× | 14.97 | **12.43** | **+2.54 PPL** |
182
+ | Mid-Band | `(0.20, 0.55, 1.15)` | 2.181 bpp | 14.7× | 30.10 | **23.08** | **+7.02 PPL** |
183
+ | **Calibrated TritQ** | **`(0.15, 0.40, 1.00)`** | **1.638 bpp** | **19.5×** | 73.75 | **66.50** | **+7.25 PPL** |
184
+ | Aggressive | `(0.10, 0.30, 0.75)` | 0.914 bpp | 35.0× | **835.92** | 1117.65 | $-281.73$ PPL |
185
+ | Ultra-Aggressive | `(0.08, 0.22, 0.50)` | 0.430 bpp | 74.4× | 5274.18 | **2890.48** | **+2383.70 PPL** |
186
+
187
+ * Roughness reduction: **$-72.1\%$** ($E_D^{\text{edges}}: 0.006178 \to 0.001724$; $E_D^{\text{numel}}: 0.012316 \to 0.003438$).
188
+ * At `2.181 bpp`, Topographic TritQ (`23.08 PPL`) outperforms both Standard TritQ (`30.10 PPL`) and per-tensor Spatial INT2 RTN (`41.67 PPL` at `2.00 bpp`).
189
+
190
+ <p align="center">
191
+ <img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig3_pareto_quantization.png" alt="Calibration Sweep and Empirical Rate-Distortion Frontier" width="95%">
192
+ </p>
193
+
194
+ > **Figure 3: Calibration Sweep and Rate–Distortion Frontier (TinyStories-10M).** (a) 5-point calibration sweep ($\lambda_{\text{topo}} \in \{0, 0.01, 5, 15, 30\}$, 2k steps) at `1.638 bpp`. (b) Converged 10k-step rate–distortion frontier across `1.638–3.774 bpp` (Preprint Table 5).
195
+
196
+ ### 2. Vision MLP on MNIST (`234.75K` parameters, 5 epochs)
197
+
198
+ | Metric | Standard MLP | Topographic MLP (`dreg`, $\lambda=15$) | Gain |
199
+ | :--- | :---: | :---: | :---: |
200
+ | FP32 Test Accuracy | **97.87%** | 97.58% | $-0.29\%$ (preserved) |
201
+ | Dirichlet Energy ($E_D$, `numel`) | 0.0219 | **0.0034** | **$-84.5\%$ roughness** |
202
+ | Low-Frequency Energy ($\rho \le 0.40$) | 27.87% | **52.84%** | **$1.90\times$ compaction** |
203
+ | Low-Pass Truncation Acc ($\rho \le 0.40$) | 30.69% | **41.41%** | **+10.72% abs.** (+11.01% rel. retention) |
204
+ | Base-3 TritQ Acc (`1.64 bpp`) | 87.40% | **88.59%** | **+1.19% accuracy** |
205
+
206
+ ---
207
+
208
+ ## Quantization Format (`.tritq`) & C11 Direct Spectral Micro-Kernel
209
+
210
+ * **Exact Base-3 Trit Packing + Lloyd-Max Band-2:** 5 balanced trits $\{-1, 0, +1\}$ pack bijectively into 1 byte ($3^5 = 243 \le 256 \Rightarrow 1.60\text{ bits/trit}$), quantized via 1D Lloyd-Max MSE-optimal scaling and decoded in $\mathcal{O}(1)$ time via a 1.25 KB L1-cache lookup table. Non-topographic parameters (`token_emb`, `lm_head`, LayerNorms) are preserved uncompressed in FP16.
211
+ * **Portable C11 Micro-Kernel (`kernel/spectral_dma_kernel.c`):**
212
+ * **Zero-copy async DMA double-buffering** (Win32 Events / POSIX pthreads) verified to `< 1e-5` float32 parity against PyTorch.
213
+ * **Direct Spectral-Domain Streaming MatVec (`c_spectral_matvec_direct`):** Computes $y = \operatorname{IDCT}_{1D}(S \cdot \operatorname{DCT}_{1D}(x))$ via radix-2 $\mathcal{O}(N \log N)$ 1D-DCT directly from `.tritq` bitstreams **without materializing $M \times N$ weights or storing DCT basis tables** — cutting `1024×256` FFN working SRAM from `8.4 MB` down to **`13.0 KB`** (`650×` reduction) and running **`~45×` faster** than full-matrix 2D-IDCT reconstruction (`1.9e-6` float32 parity).
214
+
215
+ ---
216
+
217
+ ## Reproducing Locally (CPU)
218
+
219
+ ```bash
220
+ # 1. Run the master end-to-end verification suite (99 tests + C parity + trained MLP + controls)
221
+ python examples/reproduce_all.py
222
+
223
+ # 2. Build and benchmark the native C11 DMA micro-kernel
224
+ python kernel/build_kernel.py
225
+ python examples/benchmark_c_dma.py
226
+
227
+ # 3. Train a small Topographic Transformer or MNIST MLP locally
228
+ python examples/train_topographic.py
229
+ python examples/train_mlp_dirichlet.py --dataset mnist --epochs 5
230
+ python examples/evaluate_falsification.py
231
+ ```
232
+
233
+ ---
234
+
235
+ ## Repository Structure
236
+
237
+ ```text
238
+ dirichlet-regularization/
239
+ ├── dreg/ # Core Python package
240
+ │ ├── topology.py # 2D Dirichlet energy, DirichletLoss & annealing schedulers
241
+ │ ├── spectral.py # Orthonormal 2D-DCT/IDCT & BlockDCTTiler
242
+ │ ├── quantization.py # Base3TritQuantizer & .tritq binary checkpoint format
243
+ │ ├── model.py # Reference TopographicTransformer & TopographicLinear
244
+ │ └── baselines.py # Haar random orthogonal rotation & truncated SVD controls
245
+ ├── kernel/ # Native C11 zero-copy DMA micro-kernel
246
+ │ ├── spectral_dma_kernel.c # O(1) trit LUT + Win32/POSIX async double-buffered DMA
247
+ │ ├── build_kernel.py # Cross-platform compiler wrapper (gcc/clang/cl)
248
+ │ └── Makefile
249
+ ├── examples/ # Local CPU/GPU reproducible scripts
250
+ │ ├── reproduce_all.py # End-to-end verification (tests, C parity, trained MLP, controls)
251
+ │ ├── train_topographic.py # Minimal TopographicTransformer training example
252
+ │ ├── train_mlp_dirichlet.py # Vision MLP benchmark (synthetic / MNIST)
253
+ │ ├── evaluate_falsification.py # Controlled Standard vs Topographic comparison
254
+ │ ├── evaluate_pareto_curve.py # Local rate-distortion sweep & plotting
255
+ │ ├── train_multiseed_sweep.py # Multi-seed robustness & annealing sweep
256
+ │ ├── train_gpt2_dirichlet.py # GPT-2 fine-tuning & TritQ audit
257
+ │ └── benchmark_c_dma.py # Native C kernel parity & throughput benchmark
258
+ ├── benchmarks/ # Modal cloud A10G GPU training & evaluation scripts
259
+ ├── results/ # Raw JSON logs backing paper Tables 4–5 and diagnostics
260
+ ├── paper/ # Academic preprint (LaTeX source, BibTeX, figures, PDF)
261
+ │ ├── paper-draft.pdf
262
+ │ ├── paper-draft.tex
263
+ │ ├── references.bib
264
+ │ └── figures/
265
+ ├── tests/ # 99 automated unit, integration, and C-kernel tests (96% cov)
266
+ ├── CITATION.cff
267
+ ├── pyproject.toml
268
+ └── README.md
269
+ ```
270
+
271
+ ---
272
+
273
+ ## Preprint & Citation
274
+
275
+ Read the full manuscript in [`paper/paper-draft.pdf`](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf):
276
+ > **"Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization"**
277
+ > *Mario Raúl Carbonell Martínez (Independent Researcher)*
278
+
279
+ ```bibtex
280
+ @misc{carbonell2026dirichlet,
281
+ title={Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization},
282
+ author={Carbonell Mart{\'i}nez, Mario Ra{\'u}l},
283
+ year={2026},
284
+ howpublished={\url{https://github.com/mcarbonell/dirichlet-regularization}}
285
+ }
286
+ ```
287
+
288
+ ---
289
+
290
+ ## License
291
+
292
+ This project is licensed under the MIT License — see the [LICENSE](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE) file for details.
@@ -0,0 +1,252 @@
1
+ <div align="center">
2
+
3
+ # Dirichlet Regularization (`dreg`)
4
+ ### Spatial Smoothness in Neural Weight Manifolds for 2D-DCT Spectral Compression
5
+
6
+ [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
7
+ [![PyTorch 2.0+](https://img.shields.io/badge/pytorch-2.0+-red.svg)](https://pytorch.org/)
8
+ [![CI](https://github.com/mcarbonell/dirichlet-regularization/actions/workflows/ci.yml/badge.svg)](https://github.com/mcarbonell/dirichlet-regularization/actions/workflows/ci.yml)
9
+ [![Tests](https://img.shields.io/badge/tests-99_passed-brightgreen.svg)](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
10
+ [![Coverage](https://img.shields.io/badge/coverage-96%25-brightgreen.svg)](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
11
+ [![Code Style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
12
+ [![Preprint (PDF)](https://img.shields.io/badge/Preprint-PDF-purple.svg)](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf)
13
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE)
14
+
15
+ </div>
16
+
17
+ ---
18
+
19
+ ## The Core Idea
20
+
21
+ Standard neural networks treat weight matrices as unstructured, permutation-invariant point clouds. After training with isotropic $L_2$ weight decay, their 2D spatial frequency spectrum resembles **flat white noise** — variance is spread uniformly across all frequencies, making frequency-domain compression ineffective.
22
+
23
+ **Dirichlet Regularization (`dreg`)** introduces a simple inductive bias during training: *neighboring entries on a 2D weight lattice should vary smoothly*, analogous to topographic organization in biological cortex.
24
+
25
+ Penalizing discrete 2D spatial differences breaks permutation symmetry and **concentrates spectral energy in the low-frequency harmonics of the orthonormal 2D-DCT** — up to **~72%** of Frobenius energy below $\rho \le 0.40$ on square Attention projections (`256×256`) and **52.8%** on MNIST MLPs — enabling hierarchical Base-3 Trit spectral quantization (`.tritq`) across a configurable `0.43–3.77 bpp` rate–distortion frontier.
26
+
27
+ <p align="center">
28
+ <img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig1_weight_heatmaps.png" alt="Spatial Weight Structures: Standard AdamW vs Dirichlet Regularization" width="90%">
29
+ </p>
30
+
31
+ > **Figure 1: Illustrative Spatial Weight Structures (MNIST MLP, 64×64 top-left patches).** 3-epoch MNIST MLP (`fc1` top, `fc2` bottom, unnormalized $\epsilon = 2 \times 10^{-3}$). Left: Standard AdamW yields high-frequency unstructured weights (edge-normalized $E_D = 0.0016$ on `fc1`). Right: 2D Dirichlet regularization induces smooth spatial manifolds ($E_D = 0.0001$, $-94\%$ roughness). On full TinyStories-10M Transformers (Table 2 of the preprint), mean $E_D$ drops by **$-72.1\%$** ($0.0123 \to 0.0034$ `numel`; $0.0062 \to 0.0017$ `edges`).
32
+
33
+ ---
34
+
35
+ ## The Mathematics
36
+
37
+ Standard $L_2$ weight decay penalizes **magnitude** and is strictly invariant to row/column permutations:
38
+
39
+ $$\mathcal{L}_{L_2}(W) = \frac{\gamma}{2} \|W\|_F^2$$
40
+
41
+ **Dirichlet Regularization** penalizes **2D discrete spatial differences** across the $M \times N$ weight lattice via the separable Neumann graph Laplacians $L_M \in \mathbb{R}^{M \times M}$ and $L_N \in \mathbb{R}^{N \times N}$:
42
+
43
+ $$\mathcal{E}_D^{\text{var}}(W) = \frac{1}{2MN}\!\left(\sum_{i=1}^{M-1}\sum_{j=1}^N (W_{i+1,j}-W_{i,j})^2 + \sum_{i=1}^M\sum_{j=1}^{N-1} (W_{i,j+1}-W_{i,j})^2\right) = \frac{1}{2MN}\Big(\operatorname{Tr}(W^\top L_M W) + \operatorname{Tr}(W L_N W^\top)\Big)$$
44
+
45
+ Because the eigenvectors of the 1D Neumann path-graph Laplacian are exactly the orthonormal **Type-II DCT** basis vectors, $\mathcal{E}_D$ diagonalizes in the 2D-DCT domain $S = D_M W D_N^\top$ (`numel` convention $= 2\,\mathcal{E}_D^{\text{var}}$):
46
+
47
+ $$\mathcal{E}_D^{\text{numel}}(W) = \frac{1}{MN} \sum_{k=0}^{M-1}\sum_{l=0}^{N-1} 4\!\left[\sin^2\!\left(\frac{\pi k}{2M}\right) + \sin^2\!\left(\frac{\pi l}{2N}\right)\right] S_{k,l}^2$$
48
+
49
+ High-frequency modes $(k, l)$ are penalized quadratically, concentrating weight energy into the lowest 2D-DCT harmonics.
50
+
51
+ > **Normalization note:** `dreg` provides `normalization="numel"` ($\frac{1}{MN}\sum \Delta^2 = 2\,\mathcal{E}_D^{\text{var}}$, default), `normalization="edges"` ($\approx \mathcal{E}_D^{\text{var}}$), and scale-invariant `normalization="rayleigh"` ($\frac{\sum \Delta^2}{2\|W\|_F^2} \in [0, 4]$, immune to $L_2$ magnitude shrinkage), plus `layer_weighting="param_weighted"` to balance gradient strength equally across square Attention and wide FFN layers. See [dreg/topology.py](https://github.com/mcarbonell/dirichlet-regularization/blob/main/dreg/topology.py) and §3.1 of the preprint.
52
+
53
+ <p align="center">
54
+ <img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig2_dct_energy_spectra.png" alt="2D-DCT Spectral Energy Compaction" width="90%">
55
+ </p>
56
+
57
+ > **Figure 2: 2D-DCT Cumulative Spectral Energy (TinyStories-10M, Layer 3).** Cumulative Frobenius energy vs. normalized radius $\rho \in [0, \sqrt{2}]$. Standard AdamW stays flat ($\sim 13\text{--}14\%$ below $\rho = 0.40$). Dirichlet regularization concentrates **~72%** of energy below $\rho = 0.40$ in Attention $W_q$ (`256×256`), and **~25%** in the $4\times$ larger FFN gate projection (`256×1024`) due to $1/(MN)$ per-weight gradient scaling.
58
+
59
+ ---
60
+
61
+ ## Quick Start
62
+
63
+ ### Installation
64
+
65
+ ```bash
66
+ pip install dirichlet-regularization
67
+ ```
68
+
69
+ Or from source:
70
+ ```bash
71
+ git clone https://github.com/mcarbonell/dirichlet-regularization.git
72
+ cd dirichlet-regularization
73
+ pip install -e .
74
+ ```
75
+
76
+ Optional extras:
77
+ ```bash
78
+ pip install -e ".[examples]" # torchvision + matplotlib for examples
79
+ pip install -e ".[dev]" # pytest, pytest-cov, ruff
80
+ ```
81
+
82
+ ### Apply to Any PyTorch Model (2 lines of code)
83
+
84
+ #### Option A: Universal loss wrapper (zero model modifications)
85
+ ```python
86
+ from dreg import DirichletLoss
87
+
88
+ # Automatically skips nn.Embedding, lm_head, frozen weights, and deduplicates tied weights.
89
+ # Use normalization="rayleigh" + layer_weighting="param_weighted" for scale-invariant,
90
+ # FFN-balanced smoothing (recommended: weight_decay=0.25–0.50), or classic normalization="numel"
91
+ # with weight_decay=15.0–30.0.
92
+ topo_reg = DirichletLoss(
93
+ weight_decay=0.50,
94
+ normalization="rayleigh",
95
+ layer_weighting="param_weighted",
96
+ )
97
+
98
+ # Inside your training loop:
99
+ task_loss = criterion(model(inputs), targets)
100
+ total_loss = task_loss + topo_reg(model)
101
+ total_loss.backward()
102
+ ```
103
+
104
+ #### Option B: Drop-in replacement for `nn.Linear`
105
+ ```python
106
+ import torch.nn as nn
107
+ from dreg import TopographicLinear
108
+
109
+ class MyModel(nn.Module):
110
+ def __init__(self):
111
+ super().__init__()
112
+ self.fc1 = TopographicLinear(512, 2048)
113
+ self.fc2 = TopographicLinear(2048, 10)
114
+ self.act = nn.GELU()
115
+
116
+ def forward(self, x):
117
+ return self.fc2(self.act(self.fc1(x)))
118
+
119
+ def dirichlet_loss(self):
120
+ return 0.5 * (self.fc1.dirichlet_energy() + self.fc2.dirichlet_energy())
121
+
122
+ model = MyModel()
123
+ loss = criterion(model(x), y) + 15.0 * model.dirichlet_loss()
124
+ loss.backward()
125
+ ```
126
+
127
+ > **Calibration tip:** Because `dirichlet_energy_2d` is normalized by $MN$, raw values are $\sim 10^{-3}\text{--}10^{-2}$. Use $\lambda_{\text{topo}} \in [5.0, 30.0]$ (rather than `0.01`, which is washed out by AdamW).
128
+
129
+ ---
130
+
131
+ ## Empirical Highlights
132
+
133
+ ### 1. TinyStories 10.56M Transformer (81.92M tokens, A10G GPU)
134
+
135
+ Evaluated on 25 validation batches (204,800 tokens; full results in [results/pareto_curve_tinystories_results.json](https://github.com/mcarbonell/dirichlet-regularization/blob/main/results/pareto_curve_tinystories_results.json) and Preprint Table 5):
136
+
137
+ | Quantization Regime | Radii $(r_0, r_1, r_2)$ | Linear Rate | Compression | Std PPL | Topo PPL ($\lambda=30$) | Dirichlet Gain |
138
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
139
+ | Dense FP32 (Reference) | — | 32.000 bpp | 1.0× | **6.11** | 6.46 | $-0.35$ PPL |
140
+ | Conservative | `(0.45, 0.90, 1.41)` | 3.774 bpp | 8.5× | 8.87 | **8.22** | **+0.65 PPL** |
141
+ | High-Fidelity | `(0.30, 0.70, 1.30)` | 2.794 bpp | 11.5× | 14.97 | **12.43** | **+2.54 PPL** |
142
+ | Mid-Band | `(0.20, 0.55, 1.15)` | 2.181 bpp | 14.7× | 30.10 | **23.08** | **+7.02 PPL** |
143
+ | **Calibrated TritQ** | **`(0.15, 0.40, 1.00)`** | **1.638 bpp** | **19.5×** | 73.75 | **66.50** | **+7.25 PPL** |
144
+ | Aggressive | `(0.10, 0.30, 0.75)` | 0.914 bpp | 35.0× | **835.92** | 1117.65 | $-281.73$ PPL |
145
+ | Ultra-Aggressive | `(0.08, 0.22, 0.50)` | 0.430 bpp | 74.4× | 5274.18 | **2890.48** | **+2383.70 PPL** |
146
+
147
+ * Roughness reduction: **$-72.1\%$** ($E_D^{\text{edges}}: 0.006178 \to 0.001724$; $E_D^{\text{numel}}: 0.012316 \to 0.003438$).
148
+ * At `2.181 bpp`, Topographic TritQ (`23.08 PPL`) outperforms both Standard TritQ (`30.10 PPL`) and per-tensor Spatial INT2 RTN (`41.67 PPL` at `2.00 bpp`).
149
+
150
+ <p align="center">
151
+ <img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig3_pareto_quantization.png" alt="Calibration Sweep and Empirical Rate-Distortion Frontier" width="95%">
152
+ </p>
153
+
154
+ > **Figure 3: Calibration Sweep and Rate–Distortion Frontier (TinyStories-10M).** (a) 5-point calibration sweep ($\lambda_{\text{topo}} \in \{0, 0.01, 5, 15, 30\}$, 2k steps) at `1.638 bpp`. (b) Converged 10k-step rate–distortion frontier across `1.638–3.774 bpp` (Preprint Table 5).
155
+
156
+ ### 2. Vision MLP on MNIST (`234.75K` parameters, 5 epochs)
157
+
158
+ | Metric | Standard MLP | Topographic MLP (`dreg`, $\lambda=15$) | Gain |
159
+ | :--- | :---: | :---: | :---: |
160
+ | FP32 Test Accuracy | **97.87%** | 97.58% | $-0.29\%$ (preserved) |
161
+ | Dirichlet Energy ($E_D$, `numel`) | 0.0219 | **0.0034** | **$-84.5\%$ roughness** |
162
+ | Low-Frequency Energy ($\rho \le 0.40$) | 27.87% | **52.84%** | **$1.90\times$ compaction** |
163
+ | Low-Pass Truncation Acc ($\rho \le 0.40$) | 30.69% | **41.41%** | **+10.72% abs.** (+11.01% rel. retention) |
164
+ | Base-3 TritQ Acc (`1.64 bpp`) | 87.40% | **88.59%** | **+1.19% accuracy** |
165
+
166
+ ---
167
+
168
+ ## Quantization Format (`.tritq`) & C11 Direct Spectral Micro-Kernel
169
+
170
+ * **Exact Base-3 Trit Packing + Lloyd-Max Band-2:** 5 balanced trits $\{-1, 0, +1\}$ pack bijectively into 1 byte ($3^5 = 243 \le 256 \Rightarrow 1.60\text{ bits/trit}$), quantized via 1D Lloyd-Max MSE-optimal scaling and decoded in $\mathcal{O}(1)$ time via a 1.25 KB L1-cache lookup table. Non-topographic parameters (`token_emb`, `lm_head`, LayerNorms) are preserved uncompressed in FP16.
171
+ * **Portable C11 Micro-Kernel (`kernel/spectral_dma_kernel.c`):**
172
+ * **Zero-copy async DMA double-buffering** (Win32 Events / POSIX pthreads) verified to `< 1e-5` float32 parity against PyTorch.
173
+ * **Direct Spectral-Domain Streaming MatVec (`c_spectral_matvec_direct`):** Computes $y = \operatorname{IDCT}_{1D}(S \cdot \operatorname{DCT}_{1D}(x))$ via radix-2 $\mathcal{O}(N \log N)$ 1D-DCT directly from `.tritq` bitstreams **without materializing $M \times N$ weights or storing DCT basis tables** — cutting `1024×256` FFN working SRAM from `8.4 MB` down to **`13.0 KB`** (`650×` reduction) and running **`~45×` faster** than full-matrix 2D-IDCT reconstruction (`1.9e-6` float32 parity).
174
+
175
+ ---
176
+
177
+ ## Reproducing Locally (CPU)
178
+
179
+ ```bash
180
+ # 1. Run the master end-to-end verification suite (99 tests + C parity + trained MLP + controls)
181
+ python examples/reproduce_all.py
182
+
183
+ # 2. Build and benchmark the native C11 DMA micro-kernel
184
+ python kernel/build_kernel.py
185
+ python examples/benchmark_c_dma.py
186
+
187
+ # 3. Train a small Topographic Transformer or MNIST MLP locally
188
+ python examples/train_topographic.py
189
+ python examples/train_mlp_dirichlet.py --dataset mnist --epochs 5
190
+ python examples/evaluate_falsification.py
191
+ ```
192
+
193
+ ---
194
+
195
+ ## Repository Structure
196
+
197
+ ```text
198
+ dirichlet-regularization/
199
+ ├── dreg/ # Core Python package
200
+ │ ├── topology.py # 2D Dirichlet energy, DirichletLoss & annealing schedulers
201
+ │ ├── spectral.py # Orthonormal 2D-DCT/IDCT & BlockDCTTiler
202
+ │ ├── quantization.py # Base3TritQuantizer & .tritq binary checkpoint format
203
+ │ ├── model.py # Reference TopographicTransformer & TopographicLinear
204
+ │ └── baselines.py # Haar random orthogonal rotation & truncated SVD controls
205
+ ├── kernel/ # Native C11 zero-copy DMA micro-kernel
206
+ │ ├── spectral_dma_kernel.c # O(1) trit LUT + Win32/POSIX async double-buffered DMA
207
+ │ ├── build_kernel.py # Cross-platform compiler wrapper (gcc/clang/cl)
208
+ │ └── Makefile
209
+ ├── examples/ # Local CPU/GPU reproducible scripts
210
+ │ ├── reproduce_all.py # End-to-end verification (tests, C parity, trained MLP, controls)
211
+ │ ├── train_topographic.py # Minimal TopographicTransformer training example
212
+ │ ├── train_mlp_dirichlet.py # Vision MLP benchmark (synthetic / MNIST)
213
+ │ ├── evaluate_falsification.py # Controlled Standard vs Topographic comparison
214
+ │ ├── evaluate_pareto_curve.py # Local rate-distortion sweep & plotting
215
+ │ ├── train_multiseed_sweep.py # Multi-seed robustness & annealing sweep
216
+ │ ├── train_gpt2_dirichlet.py # GPT-2 fine-tuning & TritQ audit
217
+ │ └── benchmark_c_dma.py # Native C kernel parity & throughput benchmark
218
+ ├── benchmarks/ # Modal cloud A10G GPU training & evaluation scripts
219
+ ├── results/ # Raw JSON logs backing paper Tables 4–5 and diagnostics
220
+ ├── paper/ # Academic preprint (LaTeX source, BibTeX, figures, PDF)
221
+ │ ├── paper-draft.pdf
222
+ │ ├── paper-draft.tex
223
+ │ ├── references.bib
224
+ │ └── figures/
225
+ ├── tests/ # 99 automated unit, integration, and C-kernel tests (96% cov)
226
+ ├── CITATION.cff
227
+ ├── pyproject.toml
228
+ └── README.md
229
+ ```
230
+
231
+ ---
232
+
233
+ ## Preprint & Citation
234
+
235
+ Read the full manuscript in [`paper/paper-draft.pdf`](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf):
236
+ > **"Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization"**
237
+ > *Mario Raúl Carbonell Martínez (Independent Researcher)*
238
+
239
+ ```bibtex
240
+ @misc{carbonell2026dirichlet,
241
+ title={Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization},
242
+ author={Carbonell Mart{\'i}nez, Mario Ra{\'u}l},
243
+ year={2026},
244
+ howpublished={\url{https://github.com/mcarbonell/dirichlet-regularization}}
245
+ }
246
+ ```
247
+
248
+ ---
249
+
250
+ ## License
251
+
252
+ This project is licensed under the MIT License — see the [LICENSE](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE) file for details.