dirichlet-regularization 1.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dirichlet_regularization-1.0.0/LICENSE +21 -0
- dirichlet_regularization-1.0.0/PKG-INFO +292 -0
- dirichlet_regularization-1.0.0/README.md +252 -0
- dirichlet_regularization-1.0.0/dirichlet_regularization.egg-info/PKG-INFO +292 -0
- dirichlet_regularization-1.0.0/dirichlet_regularization.egg-info/SOURCES.txt +21 -0
- dirichlet_regularization-1.0.0/dirichlet_regularization.egg-info/dependency_links.txt +1 -0
- dirichlet_regularization-1.0.0/dirichlet_regularization.egg-info/requires.txt +19 -0
- dirichlet_regularization-1.0.0/dirichlet_regularization.egg-info/top_level.txt +1 -0
- dirichlet_regularization-1.0.0/dreg/__init__.py +45 -0
- dirichlet_regularization-1.0.0/dreg/baselines.py +93 -0
- dirichlet_regularization-1.0.0/dreg/model.py +189 -0
- dirichlet_regularization-1.0.0/dreg/quantization.py +419 -0
- dirichlet_regularization-1.0.0/dreg/spectral.py +164 -0
- dirichlet_regularization-1.0.0/dreg/topology.py +338 -0
- dirichlet_regularization-1.0.0/pyproject.toml +80 -0
- dirichlet_regularization-1.0.0/setup.cfg +4 -0
- dirichlet_regularization-1.0.0/tests/test_baselines.py +60 -0
- dirichlet_regularization-1.0.0/tests/test_integration.py +280 -0
- dirichlet_regularization-1.0.0/tests/test_kernel.py +287 -0
- dirichlet_regularization-1.0.0/tests/test_model.py +178 -0
- dirichlet_regularization-1.0.0/tests/test_quantization.py +291 -0
- dirichlet_regularization-1.0.0/tests/test_spectral.py +154 -0
- dirichlet_regularization-1.0.0/tests/test_topology.py +302 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 mcarbonell / Topographic Spectral Research Group
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,292 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dirichlet-regularization
|
|
3
|
+
Version: 1.0.0
|
|
4
|
+
Summary: 2D Dirichlet spatial regularization for neural network weights: inducing topographic smoothness for 2D-DCT spectral compression and Base-3 trit quantization
|
|
5
|
+
Author-email: Mario Raúl Carbonell Martínez <marioraulcarbonell@gmail.com>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/mcarbonell/dirichlet-regularization
|
|
8
|
+
Project-URL: Repository, https://github.com/mcarbonell/dirichlet-regularization
|
|
9
|
+
Keywords: deep-learning,pytorch,regularization,dirichlet-energy,dct,quantization,model-compression
|
|
10
|
+
Classifier: Development Status :: 4 - Beta
|
|
11
|
+
Classifier: Intended Audience :: Science/Research
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: C
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
19
|
+
Classifier: Operating System :: OS Independent
|
|
20
|
+
Requires-Python: >=3.9
|
|
21
|
+
Description-Content-Type: text/markdown
|
|
22
|
+
License-File: LICENSE
|
|
23
|
+
Requires-Dist: torch>=2.0.0
|
|
24
|
+
Requires-Dist: numpy>=1.20.0
|
|
25
|
+
Provides-Extra: examples
|
|
26
|
+
Requires-Dist: torchvision>=0.15.0; extra == "examples"
|
|
27
|
+
Requires-Dist: matplotlib>=3.5.0; extra == "examples"
|
|
28
|
+
Provides-Extra: benchmarks
|
|
29
|
+
Requires-Dist: modal>=0.50.0; extra == "benchmarks"
|
|
30
|
+
Requires-Dist: datasets>=2.0.0; extra == "benchmarks"
|
|
31
|
+
Requires-Dist: tokenizers>=0.13.0; extra == "benchmarks"
|
|
32
|
+
Requires-Dist: transformers>=4.28.0; extra == "benchmarks"
|
|
33
|
+
Requires-Dist: matplotlib>=3.5.0; extra == "benchmarks"
|
|
34
|
+
Requires-Dist: tqdm>=4.65.0; extra == "benchmarks"
|
|
35
|
+
Provides-Extra: dev
|
|
36
|
+
Requires-Dist: pytest>=7.0.0; extra == "dev"
|
|
37
|
+
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
|
|
38
|
+
Requires-Dist: ruff>=0.1.0; extra == "dev"
|
|
39
|
+
Dynamic: license-file
|
|
40
|
+
|
|
41
|
+
<div align="center">
|
|
42
|
+
|
|
43
|
+
# Dirichlet Regularization (`dreg`)
|
|
44
|
+
### Spatial Smoothness in Neural Weight Manifolds for 2D-DCT Spectral Compression
|
|
45
|
+
|
|
46
|
+
[](https://www.python.org/downloads/)
|
|
47
|
+
[](https://pytorch.org/)
|
|
48
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/actions/workflows/ci.yml)
|
|
49
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
|
|
50
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
|
|
51
|
+
[](https://github.com/astral-sh/ruff)
|
|
52
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf)
|
|
53
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE)
|
|
54
|
+
|
|
55
|
+
</div>
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## The Core Idea
|
|
60
|
+
|
|
61
|
+
Standard neural networks treat weight matrices as unstructured, permutation-invariant point clouds. After training with isotropic $L_2$ weight decay, their 2D spatial frequency spectrum resembles **flat white noise** — variance is spread uniformly across all frequencies, making frequency-domain compression ineffective.
|
|
62
|
+
|
|
63
|
+
**Dirichlet Regularization (`dreg`)** introduces a simple inductive bias during training: *neighboring entries on a 2D weight lattice should vary smoothly*, analogous to topographic organization in biological cortex.
|
|
64
|
+
|
|
65
|
+
Penalizing discrete 2D spatial differences breaks permutation symmetry and **concentrates spectral energy in the low-frequency harmonics of the orthonormal 2D-DCT** — up to **~72%** of Frobenius energy below $\rho \le 0.40$ on square Attention projections (`256×256`) and **52.8%** on MNIST MLPs — enabling hierarchical Base-3 Trit spectral quantization (`.tritq`) across a configurable `0.43–3.77 bpp` rate–distortion frontier.
|
|
66
|
+
|
|
67
|
+
<p align="center">
|
|
68
|
+
<img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig1_weight_heatmaps.png" alt="Spatial Weight Structures: Standard AdamW vs Dirichlet Regularization" width="90%">
|
|
69
|
+
</p>
|
|
70
|
+
|
|
71
|
+
> **Figure 1: Illustrative Spatial Weight Structures (MNIST MLP, 64×64 top-left patches).** 3-epoch MNIST MLP (`fc1` top, `fc2` bottom, unnormalized $\epsilon = 2 \times 10^{-3}$). Left: Standard AdamW yields high-frequency unstructured weights (edge-normalized $E_D = 0.0016$ on `fc1`). Right: 2D Dirichlet regularization induces smooth spatial manifolds ($E_D = 0.0001$, $-94\%$ roughness). On full TinyStories-10M Transformers (Table 2 of the preprint), mean $E_D$ drops by **$-72.1\%$** ($0.0123 \to 0.0034$ `numel`; $0.0062 \to 0.0017$ `edges`).
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
## The Mathematics
|
|
76
|
+
|
|
77
|
+
Standard $L_2$ weight decay penalizes **magnitude** and is strictly invariant to row/column permutations:
|
|
78
|
+
|
|
79
|
+
$$\mathcal{L}_{L_2}(W) = \frac{\gamma}{2} \|W\|_F^2$$
|
|
80
|
+
|
|
81
|
+
**Dirichlet Regularization** penalizes **2D discrete spatial differences** across the $M \times N$ weight lattice via the separable Neumann graph Laplacians $L_M \in \mathbb{R}^{M \times M}$ and $L_N \in \mathbb{R}^{N \times N}$:
|
|
82
|
+
|
|
83
|
+
$$\mathcal{E}_D^{\text{var}}(W) = \frac{1}{2MN}\!\left(\sum_{i=1}^{M-1}\sum_{j=1}^N (W_{i+1,j}-W_{i,j})^2 + \sum_{i=1}^M\sum_{j=1}^{N-1} (W_{i,j+1}-W_{i,j})^2\right) = \frac{1}{2MN}\Big(\operatorname{Tr}(W^\top L_M W) + \operatorname{Tr}(W L_N W^\top)\Big)$$
|
|
84
|
+
|
|
85
|
+
Because the eigenvectors of the 1D Neumann path-graph Laplacian are exactly the orthonormal **Type-II DCT** basis vectors, $\mathcal{E}_D$ diagonalizes in the 2D-DCT domain $S = D_M W D_N^\top$ (`numel` convention $= 2\,\mathcal{E}_D^{\text{var}}$):
|
|
86
|
+
|
|
87
|
+
$$\mathcal{E}_D^{\text{numel}}(W) = \frac{1}{MN} \sum_{k=0}^{M-1}\sum_{l=0}^{N-1} 4\!\left[\sin^2\!\left(\frac{\pi k}{2M}\right) + \sin^2\!\left(\frac{\pi l}{2N}\right)\right] S_{k,l}^2$$
|
|
88
|
+
|
|
89
|
+
High-frequency modes $(k, l)$ are penalized quadratically, concentrating weight energy into the lowest 2D-DCT harmonics.
|
|
90
|
+
|
|
91
|
+
> **Normalization note:** `dreg` provides `normalization="numel"` ($\frac{1}{MN}\sum \Delta^2 = 2\,\mathcal{E}_D^{\text{var}}$, default), `normalization="edges"` ($\approx \mathcal{E}_D^{\text{var}}$), and scale-invariant `normalization="rayleigh"` ($\frac{\sum \Delta^2}{2\|W\|_F^2} \in [0, 4]$, immune to $L_2$ magnitude shrinkage), plus `layer_weighting="param_weighted"` to balance gradient strength equally across square Attention and wide FFN layers. See [dreg/topology.py](https://github.com/mcarbonell/dirichlet-regularization/blob/main/dreg/topology.py) and §3.1 of the preprint.
|
|
92
|
+
|
|
93
|
+
<p align="center">
|
|
94
|
+
<img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig2_dct_energy_spectra.png" alt="2D-DCT Spectral Energy Compaction" width="90%">
|
|
95
|
+
</p>
|
|
96
|
+
|
|
97
|
+
> **Figure 2: 2D-DCT Cumulative Spectral Energy (TinyStories-10M, Layer 3).** Cumulative Frobenius energy vs. normalized radius $\rho \in [0, \sqrt{2}]$. Standard AdamW stays flat ($\sim 13\text{--}14\%$ below $\rho = 0.40$). Dirichlet regularization concentrates **~72%** of energy below $\rho = 0.40$ in Attention $W_q$ (`256×256`), and **~25%** in the $4\times$ larger FFN gate projection (`256×1024`) due to $1/(MN)$ per-weight gradient scaling.
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## Quick Start
|
|
102
|
+
|
|
103
|
+
### Installation
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
pip install dirichlet-regularization
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
Or from source:
|
|
110
|
+
```bash
|
|
111
|
+
git clone https://github.com/mcarbonell/dirichlet-regularization.git
|
|
112
|
+
cd dirichlet-regularization
|
|
113
|
+
pip install -e .
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Optional extras:
|
|
117
|
+
```bash
|
|
118
|
+
pip install -e ".[examples]" # torchvision + matplotlib for examples
|
|
119
|
+
pip install -e ".[dev]" # pytest, pytest-cov, ruff
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
### Apply to Any PyTorch Model (2 lines of code)
|
|
123
|
+
|
|
124
|
+
#### Option A: Universal loss wrapper (zero model modifications)
|
|
125
|
+
```python
|
|
126
|
+
from dreg import DirichletLoss
|
|
127
|
+
|
|
128
|
+
# Automatically skips nn.Embedding, lm_head, frozen weights, and deduplicates tied weights.
|
|
129
|
+
# Use normalization="rayleigh" + layer_weighting="param_weighted" for scale-invariant,
|
|
130
|
+
# FFN-balanced smoothing (recommended: weight_decay=0.25–0.50), or classic normalization="numel"
|
|
131
|
+
# with weight_decay=15.0–30.0.
|
|
132
|
+
topo_reg = DirichletLoss(
|
|
133
|
+
weight_decay=0.50,
|
|
134
|
+
normalization="rayleigh",
|
|
135
|
+
layer_weighting="param_weighted",
|
|
136
|
+
)
|
|
137
|
+
|
|
138
|
+
# Inside your training loop:
|
|
139
|
+
task_loss = criterion(model(inputs), targets)
|
|
140
|
+
total_loss = task_loss + topo_reg(model)
|
|
141
|
+
total_loss.backward()
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
#### Option B: Drop-in replacement for `nn.Linear`
|
|
145
|
+
```python
|
|
146
|
+
import torch.nn as nn
|
|
147
|
+
from dreg import TopographicLinear
|
|
148
|
+
|
|
149
|
+
class MyModel(nn.Module):
|
|
150
|
+
def __init__(self):
|
|
151
|
+
super().__init__()
|
|
152
|
+
self.fc1 = TopographicLinear(512, 2048)
|
|
153
|
+
self.fc2 = TopographicLinear(2048, 10)
|
|
154
|
+
self.act = nn.GELU()
|
|
155
|
+
|
|
156
|
+
def forward(self, x):
|
|
157
|
+
return self.fc2(self.act(self.fc1(x)))
|
|
158
|
+
|
|
159
|
+
def dirichlet_loss(self):
|
|
160
|
+
return 0.5 * (self.fc1.dirichlet_energy() + self.fc2.dirichlet_energy())
|
|
161
|
+
|
|
162
|
+
model = MyModel()
|
|
163
|
+
loss = criterion(model(x), y) + 15.0 * model.dirichlet_loss()
|
|
164
|
+
loss.backward()
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
> **Calibration tip:** Because `dirichlet_energy_2d` is normalized by $MN$, raw values are $\sim 10^{-3}\text{--}10^{-2}$. Use $\lambda_{\text{topo}} \in [5.0, 30.0]$ (rather than `0.01`, which is washed out by AdamW).
|
|
168
|
+
|
|
169
|
+
---
|
|
170
|
+
|
|
171
|
+
## Empirical Highlights
|
|
172
|
+
|
|
173
|
+
### 1. TinyStories 10.56M Transformer (81.92M tokens, A10G GPU)
|
|
174
|
+
|
|
175
|
+
Evaluated on 25 validation batches (204,800 tokens; full results in [results/pareto_curve_tinystories_results.json](https://github.com/mcarbonell/dirichlet-regularization/blob/main/results/pareto_curve_tinystories_results.json) and Preprint Table 5):
|
|
176
|
+
|
|
177
|
+
| Quantization Regime | Radii $(r_0, r_1, r_2)$ | Linear Rate | Compression | Std PPL | Topo PPL ($\lambda=30$) | Dirichlet Gain |
|
|
178
|
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
179
|
+
| Dense FP32 (Reference) | — | 32.000 bpp | 1.0× | **6.11** | 6.46 | $-0.35$ PPL |
|
|
180
|
+
| Conservative | `(0.45, 0.90, 1.41)` | 3.774 bpp | 8.5× | 8.87 | **8.22** | **+0.65 PPL** |
|
|
181
|
+
| High-Fidelity | `(0.30, 0.70, 1.30)` | 2.794 bpp | 11.5× | 14.97 | **12.43** | **+2.54 PPL** |
|
|
182
|
+
| Mid-Band | `(0.20, 0.55, 1.15)` | 2.181 bpp | 14.7× | 30.10 | **23.08** | **+7.02 PPL** |
|
|
183
|
+
| **Calibrated TritQ** | **`(0.15, 0.40, 1.00)`** | **1.638 bpp** | **19.5×** | 73.75 | **66.50** | **+7.25 PPL** |
|
|
184
|
+
| Aggressive | `(0.10, 0.30, 0.75)` | 0.914 bpp | 35.0× | **835.92** | 1117.65 | $-281.73$ PPL |
|
|
185
|
+
| Ultra-Aggressive | `(0.08, 0.22, 0.50)` | 0.430 bpp | 74.4× | 5274.18 | **2890.48** | **+2383.70 PPL** |
|
|
186
|
+
|
|
187
|
+
* Roughness reduction: **$-72.1\%$** ($E_D^{\text{edges}}: 0.006178 \to 0.001724$; $E_D^{\text{numel}}: 0.012316 \to 0.003438$).
|
|
188
|
+
* At `2.181 bpp`, Topographic TritQ (`23.08 PPL`) outperforms both Standard TritQ (`30.10 PPL`) and per-tensor Spatial INT2 RTN (`41.67 PPL` at `2.00 bpp`).
|
|
189
|
+
|
|
190
|
+
<p align="center">
|
|
191
|
+
<img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig3_pareto_quantization.png" alt="Calibration Sweep and Empirical Rate-Distortion Frontier" width="95%">
|
|
192
|
+
</p>
|
|
193
|
+
|
|
194
|
+
> **Figure 3: Calibration Sweep and Rate–Distortion Frontier (TinyStories-10M).** (a) 5-point calibration sweep ($\lambda_{\text{topo}} \in \{0, 0.01, 5, 15, 30\}$, 2k steps) at `1.638 bpp`. (b) Converged 10k-step rate–distortion frontier across `1.638–3.774 bpp` (Preprint Table 5).
|
|
195
|
+
|
|
196
|
+
### 2. Vision MLP on MNIST (`234.75K` parameters, 5 epochs)
|
|
197
|
+
|
|
198
|
+
| Metric | Standard MLP | Topographic MLP (`dreg`, $\lambda=15$) | Gain |
|
|
199
|
+
| :--- | :---: | :---: | :---: |
|
|
200
|
+
| FP32 Test Accuracy | **97.87%** | 97.58% | $-0.29\%$ (preserved) |
|
|
201
|
+
| Dirichlet Energy ($E_D$, `numel`) | 0.0219 | **0.0034** | **$-84.5\%$ roughness** |
|
|
202
|
+
| Low-Frequency Energy ($\rho \le 0.40$) | 27.87% | **52.84%** | **$1.90\times$ compaction** |
|
|
203
|
+
| Low-Pass Truncation Acc ($\rho \le 0.40$) | 30.69% | **41.41%** | **+10.72% abs.** (+11.01% rel. retention) |
|
|
204
|
+
| Base-3 TritQ Acc (`1.64 bpp`) | 87.40% | **88.59%** | **+1.19% accuracy** |
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## Quantization Format (`.tritq`) & C11 Direct Spectral Micro-Kernel
|
|
209
|
+
|
|
210
|
+
* **Exact Base-3 Trit Packing + Lloyd-Max Band-2:** 5 balanced trits $\{-1, 0, +1\}$ pack bijectively into 1 byte ($3^5 = 243 \le 256 \Rightarrow 1.60\text{ bits/trit}$), quantized via 1D Lloyd-Max MSE-optimal scaling and decoded in $\mathcal{O}(1)$ time via a 1.25 KB L1-cache lookup table. Non-topographic parameters (`token_emb`, `lm_head`, LayerNorms) are preserved uncompressed in FP16.
|
|
211
|
+
* **Portable C11 Micro-Kernel (`kernel/spectral_dma_kernel.c`):**
|
|
212
|
+
* **Zero-copy async DMA double-buffering** (Win32 Events / POSIX pthreads) verified to `< 1e-5` float32 parity against PyTorch.
|
|
213
|
+
* **Direct Spectral-Domain Streaming MatVec (`c_spectral_matvec_direct`):** Computes $y = \operatorname{IDCT}_{1D}(S \cdot \operatorname{DCT}_{1D}(x))$ via radix-2 $\mathcal{O}(N \log N)$ 1D-DCT directly from `.tritq` bitstreams **without materializing $M \times N$ weights or storing DCT basis tables** — cutting `1024×256` FFN working SRAM from `8.4 MB` down to **`13.0 KB`** (`650×` reduction) and running **`~45×` faster** than full-matrix 2D-IDCT reconstruction (`1.9e-6` float32 parity).
|
|
214
|
+
|
|
215
|
+
---
|
|
216
|
+
|
|
217
|
+
## Reproducing Locally (CPU)
|
|
218
|
+
|
|
219
|
+
```bash
|
|
220
|
+
# 1. Run the master end-to-end verification suite (99 tests + C parity + trained MLP + controls)
|
|
221
|
+
python examples/reproduce_all.py
|
|
222
|
+
|
|
223
|
+
# 2. Build and benchmark the native C11 DMA micro-kernel
|
|
224
|
+
python kernel/build_kernel.py
|
|
225
|
+
python examples/benchmark_c_dma.py
|
|
226
|
+
|
|
227
|
+
# 3. Train a small Topographic Transformer or MNIST MLP locally
|
|
228
|
+
python examples/train_topographic.py
|
|
229
|
+
python examples/train_mlp_dirichlet.py --dataset mnist --epochs 5
|
|
230
|
+
python examples/evaluate_falsification.py
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
---
|
|
234
|
+
|
|
235
|
+
## Repository Structure
|
|
236
|
+
|
|
237
|
+
```text
|
|
238
|
+
dirichlet-regularization/
|
|
239
|
+
├── dreg/ # Core Python package
|
|
240
|
+
│ ├── topology.py # 2D Dirichlet energy, DirichletLoss & annealing schedulers
|
|
241
|
+
│ ├── spectral.py # Orthonormal 2D-DCT/IDCT & BlockDCTTiler
|
|
242
|
+
│ ├── quantization.py # Base3TritQuantizer & .tritq binary checkpoint format
|
|
243
|
+
│ ├── model.py # Reference TopographicTransformer & TopographicLinear
|
|
244
|
+
│ └── baselines.py # Haar random orthogonal rotation & truncated SVD controls
|
|
245
|
+
├── kernel/ # Native C11 zero-copy DMA micro-kernel
|
|
246
|
+
│ ├── spectral_dma_kernel.c # O(1) trit LUT + Win32/POSIX async double-buffered DMA
|
|
247
|
+
│ ├── build_kernel.py # Cross-platform compiler wrapper (gcc/clang/cl)
|
|
248
|
+
│ └── Makefile
|
|
249
|
+
├── examples/ # Local CPU/GPU reproducible scripts
|
|
250
|
+
│ ├── reproduce_all.py # End-to-end verification (tests, C parity, trained MLP, controls)
|
|
251
|
+
│ ├── train_topographic.py # Minimal TopographicTransformer training example
|
|
252
|
+
│ ├── train_mlp_dirichlet.py # Vision MLP benchmark (synthetic / MNIST)
|
|
253
|
+
│ ├── evaluate_falsification.py # Controlled Standard vs Topographic comparison
|
|
254
|
+
│ ├── evaluate_pareto_curve.py # Local rate-distortion sweep & plotting
|
|
255
|
+
│ ├── train_multiseed_sweep.py # Multi-seed robustness & annealing sweep
|
|
256
|
+
│ ├── train_gpt2_dirichlet.py # GPT-2 fine-tuning & TritQ audit
|
|
257
|
+
│ └── benchmark_c_dma.py # Native C kernel parity & throughput benchmark
|
|
258
|
+
├── benchmarks/ # Modal cloud A10G GPU training & evaluation scripts
|
|
259
|
+
├── results/ # Raw JSON logs backing paper Tables 4–5 and diagnostics
|
|
260
|
+
├── paper/ # Academic preprint (LaTeX source, BibTeX, figures, PDF)
|
|
261
|
+
│ ├── paper-draft.pdf
|
|
262
|
+
│ ├── paper-draft.tex
|
|
263
|
+
│ ├── references.bib
|
|
264
|
+
│ └── figures/
|
|
265
|
+
├── tests/ # 99 automated unit, integration, and C-kernel tests (96% cov)
|
|
266
|
+
├── CITATION.cff
|
|
267
|
+
├── pyproject.toml
|
|
268
|
+
└── README.md
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
---
|
|
272
|
+
|
|
273
|
+
## Preprint & Citation
|
|
274
|
+
|
|
275
|
+
Read the full manuscript in [`paper/paper-draft.pdf`](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf):
|
|
276
|
+
> **"Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization"**
|
|
277
|
+
> *Mario Raúl Carbonell Martínez (Independent Researcher)*
|
|
278
|
+
|
|
279
|
+
```bibtex
|
|
280
|
+
@misc{carbonell2026dirichlet,
|
|
281
|
+
title={Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization},
|
|
282
|
+
author={Carbonell Mart{\'i}nez, Mario Ra{\'u}l},
|
|
283
|
+
year={2026},
|
|
284
|
+
howpublished={\url{https://github.com/mcarbonell/dirichlet-regularization}}
|
|
285
|
+
}
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## License
|
|
291
|
+
|
|
292
|
+
This project is licensed under the MIT License — see the [LICENSE](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE) file for details.
|
|
@@ -0,0 +1,252 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
3
|
+
# Dirichlet Regularization (`dreg`)
|
|
4
|
+
### Spatial Smoothness in Neural Weight Manifolds for 2D-DCT Spectral Compression
|
|
5
|
+
|
|
6
|
+
[](https://www.python.org/downloads/)
|
|
7
|
+
[](https://pytorch.org/)
|
|
8
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/actions/workflows/ci.yml)
|
|
9
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
|
|
10
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/tree/main/tests)
|
|
11
|
+
[](https://github.com/astral-sh/ruff)
|
|
12
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf)
|
|
13
|
+
[](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE)
|
|
14
|
+
|
|
15
|
+
</div>
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## The Core Idea
|
|
20
|
+
|
|
21
|
+
Standard neural networks treat weight matrices as unstructured, permutation-invariant point clouds. After training with isotropic $L_2$ weight decay, their 2D spatial frequency spectrum resembles **flat white noise** — variance is spread uniformly across all frequencies, making frequency-domain compression ineffective.
|
|
22
|
+
|
|
23
|
+
**Dirichlet Regularization (`dreg`)** introduces a simple inductive bias during training: *neighboring entries on a 2D weight lattice should vary smoothly*, analogous to topographic organization in biological cortex.
|
|
24
|
+
|
|
25
|
+
Penalizing discrete 2D spatial differences breaks permutation symmetry and **concentrates spectral energy in the low-frequency harmonics of the orthonormal 2D-DCT** — up to **~72%** of Frobenius energy below $\rho \le 0.40$ on square Attention projections (`256×256`) and **52.8%** on MNIST MLPs — enabling hierarchical Base-3 Trit spectral quantization (`.tritq`) across a configurable `0.43–3.77 bpp` rate–distortion frontier.
|
|
26
|
+
|
|
27
|
+
<p align="center">
|
|
28
|
+
<img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig1_weight_heatmaps.png" alt="Spatial Weight Structures: Standard AdamW vs Dirichlet Regularization" width="90%">
|
|
29
|
+
</p>
|
|
30
|
+
|
|
31
|
+
> **Figure 1: Illustrative Spatial Weight Structures (MNIST MLP, 64×64 top-left patches).** 3-epoch MNIST MLP (`fc1` top, `fc2` bottom, unnormalized $\epsilon = 2 \times 10^{-3}$). Left: Standard AdamW yields high-frequency unstructured weights (edge-normalized $E_D = 0.0016$ on `fc1`). Right: 2D Dirichlet regularization induces smooth spatial manifolds ($E_D = 0.0001$, $-94\%$ roughness). On full TinyStories-10M Transformers (Table 2 of the preprint), mean $E_D$ drops by **$-72.1\%$** ($0.0123 \to 0.0034$ `numel`; $0.0062 \to 0.0017$ `edges`).
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## The Mathematics
|
|
36
|
+
|
|
37
|
+
Standard $L_2$ weight decay penalizes **magnitude** and is strictly invariant to row/column permutations:
|
|
38
|
+
|
|
39
|
+
$$\mathcal{L}_{L_2}(W) = \frac{\gamma}{2} \|W\|_F^2$$
|
|
40
|
+
|
|
41
|
+
**Dirichlet Regularization** penalizes **2D discrete spatial differences** across the $M \times N$ weight lattice via the separable Neumann graph Laplacians $L_M \in \mathbb{R}^{M \times M}$ and $L_N \in \mathbb{R}^{N \times N}$:
|
|
42
|
+
|
|
43
|
+
$$\mathcal{E}_D^{\text{var}}(W) = \frac{1}{2MN}\!\left(\sum_{i=1}^{M-1}\sum_{j=1}^N (W_{i+1,j}-W_{i,j})^2 + \sum_{i=1}^M\sum_{j=1}^{N-1} (W_{i,j+1}-W_{i,j})^2\right) = \frac{1}{2MN}\Big(\operatorname{Tr}(W^\top L_M W) + \operatorname{Tr}(W L_N W^\top)\Big)$$
|
|
44
|
+
|
|
45
|
+
Because the eigenvectors of the 1D Neumann path-graph Laplacian are exactly the orthonormal **Type-II DCT** basis vectors, $\mathcal{E}_D$ diagonalizes in the 2D-DCT domain $S = D_M W D_N^\top$ (`numel` convention $= 2\,\mathcal{E}_D^{\text{var}}$):
|
|
46
|
+
|
|
47
|
+
$$\mathcal{E}_D^{\text{numel}}(W) = \frac{1}{MN} \sum_{k=0}^{M-1}\sum_{l=0}^{N-1} 4\!\left[\sin^2\!\left(\frac{\pi k}{2M}\right) + \sin^2\!\left(\frac{\pi l}{2N}\right)\right] S_{k,l}^2$$
|
|
48
|
+
|
|
49
|
+
High-frequency modes $(k, l)$ are penalized quadratically, concentrating weight energy into the lowest 2D-DCT harmonics.
|
|
50
|
+
|
|
51
|
+
> **Normalization note:** `dreg` provides `normalization="numel"` ($\frac{1}{MN}\sum \Delta^2 = 2\,\mathcal{E}_D^{\text{var}}$, default), `normalization="edges"` ($\approx \mathcal{E}_D^{\text{var}}$), and scale-invariant `normalization="rayleigh"` ($\frac{\sum \Delta^2}{2\|W\|_F^2} \in [0, 4]$, immune to $L_2$ magnitude shrinkage), plus `layer_weighting="param_weighted"` to balance gradient strength equally across square Attention and wide FFN layers. See [dreg/topology.py](https://github.com/mcarbonell/dirichlet-regularization/blob/main/dreg/topology.py) and §3.1 of the preprint.
|
|
52
|
+
|
|
53
|
+
<p align="center">
|
|
54
|
+
<img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig2_dct_energy_spectra.png" alt="2D-DCT Spectral Energy Compaction" width="90%">
|
|
55
|
+
</p>
|
|
56
|
+
|
|
57
|
+
> **Figure 2: 2D-DCT Cumulative Spectral Energy (TinyStories-10M, Layer 3).** Cumulative Frobenius energy vs. normalized radius $\rho \in [0, \sqrt{2}]$. Standard AdamW stays flat ($\sim 13\text{--}14\%$ below $\rho = 0.40$). Dirichlet regularization concentrates **~72%** of energy below $\rho = 0.40$ in Attention $W_q$ (`256×256`), and **~25%** in the $4\times$ larger FFN gate projection (`256×1024`) due to $1/(MN)$ per-weight gradient scaling.
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Quick Start
|
|
62
|
+
|
|
63
|
+
### Installation
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
pip install dirichlet-regularization
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Or from source:
|
|
70
|
+
```bash
|
|
71
|
+
git clone https://github.com/mcarbonell/dirichlet-regularization.git
|
|
72
|
+
cd dirichlet-regularization
|
|
73
|
+
pip install -e .
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
Optional extras:
|
|
77
|
+
```bash
|
|
78
|
+
pip install -e ".[examples]" # torchvision + matplotlib for examples
|
|
79
|
+
pip install -e ".[dev]" # pytest, pytest-cov, ruff
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
### Apply to Any PyTorch Model (2 lines of code)
|
|
83
|
+
|
|
84
|
+
#### Option A: Universal loss wrapper (zero model modifications)
|
|
85
|
+
```python
|
|
86
|
+
from dreg import DirichletLoss
|
|
87
|
+
|
|
88
|
+
# Automatically skips nn.Embedding, lm_head, frozen weights, and deduplicates tied weights.
|
|
89
|
+
# Use normalization="rayleigh" + layer_weighting="param_weighted" for scale-invariant,
|
|
90
|
+
# FFN-balanced smoothing (recommended: weight_decay=0.25–0.50), or classic normalization="numel"
|
|
91
|
+
# with weight_decay=15.0–30.0.
|
|
92
|
+
topo_reg = DirichletLoss(
|
|
93
|
+
weight_decay=0.50,
|
|
94
|
+
normalization="rayleigh",
|
|
95
|
+
layer_weighting="param_weighted",
|
|
96
|
+
)
|
|
97
|
+
|
|
98
|
+
# Inside your training loop:
|
|
99
|
+
task_loss = criterion(model(inputs), targets)
|
|
100
|
+
total_loss = task_loss + topo_reg(model)
|
|
101
|
+
total_loss.backward()
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
#### Option B: Drop-in replacement for `nn.Linear`
|
|
105
|
+
```python
|
|
106
|
+
import torch.nn as nn
|
|
107
|
+
from dreg import TopographicLinear
|
|
108
|
+
|
|
109
|
+
class MyModel(nn.Module):
|
|
110
|
+
def __init__(self):
|
|
111
|
+
super().__init__()
|
|
112
|
+
self.fc1 = TopographicLinear(512, 2048)
|
|
113
|
+
self.fc2 = TopographicLinear(2048, 10)
|
|
114
|
+
self.act = nn.GELU()
|
|
115
|
+
|
|
116
|
+
def forward(self, x):
|
|
117
|
+
return self.fc2(self.act(self.fc1(x)))
|
|
118
|
+
|
|
119
|
+
def dirichlet_loss(self):
|
|
120
|
+
return 0.5 * (self.fc1.dirichlet_energy() + self.fc2.dirichlet_energy())
|
|
121
|
+
|
|
122
|
+
model = MyModel()
|
|
123
|
+
loss = criterion(model(x), y) + 15.0 * model.dirichlet_loss()
|
|
124
|
+
loss.backward()
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
> **Calibration tip:** Because `dirichlet_energy_2d` is normalized by $MN$, raw values are $\sim 10^{-3}\text{--}10^{-2}$. Use $\lambda_{\text{topo}} \in [5.0, 30.0]$ (rather than `0.01`, which is washed out by AdamW).
|
|
128
|
+
|
|
129
|
+
---
|
|
130
|
+
|
|
131
|
+
## Empirical Highlights
|
|
132
|
+
|
|
133
|
+
### 1. TinyStories 10.56M Transformer (81.92M tokens, A10G GPU)
|
|
134
|
+
|
|
135
|
+
Evaluated on 25 validation batches (204,800 tokens; full results in [results/pareto_curve_tinystories_results.json](https://github.com/mcarbonell/dirichlet-regularization/blob/main/results/pareto_curve_tinystories_results.json) and Preprint Table 5):
|
|
136
|
+
|
|
137
|
+
| Quantization Regime | Radii $(r_0, r_1, r_2)$ | Linear Rate | Compression | Std PPL | Topo PPL ($\lambda=30$) | Dirichlet Gain |
|
|
138
|
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
139
|
+
| Dense FP32 (Reference) | — | 32.000 bpp | 1.0× | **6.11** | 6.46 | $-0.35$ PPL |
|
|
140
|
+
| Conservative | `(0.45, 0.90, 1.41)` | 3.774 bpp | 8.5× | 8.87 | **8.22** | **+0.65 PPL** |
|
|
141
|
+
| High-Fidelity | `(0.30, 0.70, 1.30)` | 2.794 bpp | 11.5× | 14.97 | **12.43** | **+2.54 PPL** |
|
|
142
|
+
| Mid-Band | `(0.20, 0.55, 1.15)` | 2.181 bpp | 14.7× | 30.10 | **23.08** | **+7.02 PPL** |
|
|
143
|
+
| **Calibrated TritQ** | **`(0.15, 0.40, 1.00)`** | **1.638 bpp** | **19.5×** | 73.75 | **66.50** | **+7.25 PPL** |
|
|
144
|
+
| Aggressive | `(0.10, 0.30, 0.75)` | 0.914 bpp | 35.0× | **835.92** | 1117.65 | $-281.73$ PPL |
|
|
145
|
+
| Ultra-Aggressive | `(0.08, 0.22, 0.50)` | 0.430 bpp | 74.4× | 5274.18 | **2890.48** | **+2383.70 PPL** |
|
|
146
|
+
|
|
147
|
+
* Roughness reduction: **$-72.1\%$** ($E_D^{\text{edges}}: 0.006178 \to 0.001724$; $E_D^{\text{numel}}: 0.012316 \to 0.003438$).
|
|
148
|
+
* At `2.181 bpp`, Topographic TritQ (`23.08 PPL`) outperforms both Standard TritQ (`30.10 PPL`) and per-tensor Spatial INT2 RTN (`41.67 PPL` at `2.00 bpp`).
|
|
149
|
+
|
|
150
|
+
<p align="center">
|
|
151
|
+
<img src="https://raw.githubusercontent.com/mcarbonell/dirichlet-regularization/main/docs/figures/fig3_pareto_quantization.png" alt="Calibration Sweep and Empirical Rate-Distortion Frontier" width="95%">
|
|
152
|
+
</p>
|
|
153
|
+
|
|
154
|
+
> **Figure 3: Calibration Sweep and Rate–Distortion Frontier (TinyStories-10M).** (a) 5-point calibration sweep ($\lambda_{\text{topo}} \in \{0, 0.01, 5, 15, 30\}$, 2k steps) at `1.638 bpp`. (b) Converged 10k-step rate–distortion frontier across `1.638–3.774 bpp` (Preprint Table 5).
|
|
155
|
+
|
|
156
|
+
### 2. Vision MLP on MNIST (`234.75K` parameters, 5 epochs)
|
|
157
|
+
|
|
158
|
+
| Metric | Standard MLP | Topographic MLP (`dreg`, $\lambda=15$) | Gain |
|
|
159
|
+
| :--- | :---: | :---: | :---: |
|
|
160
|
+
| FP32 Test Accuracy | **97.87%** | 97.58% | $-0.29\%$ (preserved) |
|
|
161
|
+
| Dirichlet Energy ($E_D$, `numel`) | 0.0219 | **0.0034** | **$-84.5\%$ roughness** |
|
|
162
|
+
| Low-Frequency Energy ($\rho \le 0.40$) | 27.87% | **52.84%** | **$1.90\times$ compaction** |
|
|
163
|
+
| Low-Pass Truncation Acc ($\rho \le 0.40$) | 30.69% | **41.41%** | **+10.72% abs.** (+11.01% rel. retention) |
|
|
164
|
+
| Base-3 TritQ Acc (`1.64 bpp`) | 87.40% | **88.59%** | **+1.19% accuracy** |
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## Quantization Format (`.tritq`) & C11 Direct Spectral Micro-Kernel
|
|
169
|
+
|
|
170
|
+
* **Exact Base-3 Trit Packing + Lloyd-Max Band-2:** 5 balanced trits $\{-1, 0, +1\}$ pack bijectively into 1 byte ($3^5 = 243 \le 256 \Rightarrow 1.60\text{ bits/trit}$), quantized via 1D Lloyd-Max MSE-optimal scaling and decoded in $\mathcal{O}(1)$ time via a 1.25 KB L1-cache lookup table. Non-topographic parameters (`token_emb`, `lm_head`, LayerNorms) are preserved uncompressed in FP16.
|
|
171
|
+
* **Portable C11 Micro-Kernel (`kernel/spectral_dma_kernel.c`):**
|
|
172
|
+
* **Zero-copy async DMA double-buffering** (Win32 Events / POSIX pthreads) verified to `< 1e-5` float32 parity against PyTorch.
|
|
173
|
+
* **Direct Spectral-Domain Streaming MatVec (`c_spectral_matvec_direct`):** Computes $y = \operatorname{IDCT}_{1D}(S \cdot \operatorname{DCT}_{1D}(x))$ via radix-2 $\mathcal{O}(N \log N)$ 1D-DCT directly from `.tritq` bitstreams **without materializing $M \times N$ weights or storing DCT basis tables** — cutting `1024×256` FFN working SRAM from `8.4 MB` down to **`13.0 KB`** (`650×` reduction) and running **`~45×` faster** than full-matrix 2D-IDCT reconstruction (`1.9e-6` float32 parity).
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## Reproducing Locally (CPU)
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
# 1. Run the master end-to-end verification suite (99 tests + C parity + trained MLP + controls)
|
|
181
|
+
python examples/reproduce_all.py
|
|
182
|
+
|
|
183
|
+
# 2. Build and benchmark the native C11 DMA micro-kernel
|
|
184
|
+
python kernel/build_kernel.py
|
|
185
|
+
python examples/benchmark_c_dma.py
|
|
186
|
+
|
|
187
|
+
# 3. Train a small Topographic Transformer or MNIST MLP locally
|
|
188
|
+
python examples/train_topographic.py
|
|
189
|
+
python examples/train_mlp_dirichlet.py --dataset mnist --epochs 5
|
|
190
|
+
python examples/evaluate_falsification.py
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
---
|
|
194
|
+
|
|
195
|
+
## Repository Structure
|
|
196
|
+
|
|
197
|
+
```text
|
|
198
|
+
dirichlet-regularization/
|
|
199
|
+
├── dreg/ # Core Python package
|
|
200
|
+
│ ├── topology.py # 2D Dirichlet energy, DirichletLoss & annealing schedulers
|
|
201
|
+
│ ├── spectral.py # Orthonormal 2D-DCT/IDCT & BlockDCTTiler
|
|
202
|
+
│ ├── quantization.py # Base3TritQuantizer & .tritq binary checkpoint format
|
|
203
|
+
│ ├── model.py # Reference TopographicTransformer & TopographicLinear
|
|
204
|
+
│ └── baselines.py # Haar random orthogonal rotation & truncated SVD controls
|
|
205
|
+
├── kernel/ # Native C11 zero-copy DMA micro-kernel
|
|
206
|
+
│ ├── spectral_dma_kernel.c # O(1) trit LUT + Win32/POSIX async double-buffered DMA
|
|
207
|
+
│ ├── build_kernel.py # Cross-platform compiler wrapper (gcc/clang/cl)
|
|
208
|
+
│ └── Makefile
|
|
209
|
+
├── examples/ # Local CPU/GPU reproducible scripts
|
|
210
|
+
│ ├── reproduce_all.py # End-to-end verification (tests, C parity, trained MLP, controls)
|
|
211
|
+
│ ├── train_topographic.py # Minimal TopographicTransformer training example
|
|
212
|
+
│ ├── train_mlp_dirichlet.py # Vision MLP benchmark (synthetic / MNIST)
|
|
213
|
+
│ ├── evaluate_falsification.py # Controlled Standard vs Topographic comparison
|
|
214
|
+
│ ├── evaluate_pareto_curve.py # Local rate-distortion sweep & plotting
|
|
215
|
+
│ ├── train_multiseed_sweep.py # Multi-seed robustness & annealing sweep
|
|
216
|
+
│ ├── train_gpt2_dirichlet.py # GPT-2 fine-tuning & TritQ audit
|
|
217
|
+
│ └── benchmark_c_dma.py # Native C kernel parity & throughput benchmark
|
|
218
|
+
├── benchmarks/ # Modal cloud A10G GPU training & evaluation scripts
|
|
219
|
+
├── results/ # Raw JSON logs backing paper Tables 4–5 and diagnostics
|
|
220
|
+
├── paper/ # Academic preprint (LaTeX source, BibTeX, figures, PDF)
|
|
221
|
+
│ ├── paper-draft.pdf
|
|
222
|
+
│ ├── paper-draft.tex
|
|
223
|
+
│ ├── references.bib
|
|
224
|
+
│ └── figures/
|
|
225
|
+
├── tests/ # 99 automated unit, integration, and C-kernel tests (96% cov)
|
|
226
|
+
├── CITATION.cff
|
|
227
|
+
├── pyproject.toml
|
|
228
|
+
└── README.md
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
---
|
|
232
|
+
|
|
233
|
+
## Preprint & Citation
|
|
234
|
+
|
|
235
|
+
Read the full manuscript in [`paper/paper-draft.pdf`](https://github.com/mcarbonell/dirichlet-regularization/blob/main/paper/paper-draft.pdf):
|
|
236
|
+
> **"Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization"**
|
|
237
|
+
> *Mario Raúl Carbonell Martínez (Independent Researcher)*
|
|
238
|
+
|
|
239
|
+
```bibtex
|
|
240
|
+
@misc{carbonell2026dirichlet,
|
|
241
|
+
title={Inducing Spectral Smoothness in Neural Weight Manifolds via 2D Dirichlet Regularization},
|
|
242
|
+
author={Carbonell Mart{\'i}nez, Mario Ra{\'u}l},
|
|
243
|
+
year={2026},
|
|
244
|
+
howpublished={\url{https://github.com/mcarbonell/dirichlet-regularization}}
|
|
245
|
+
}
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
---
|
|
249
|
+
|
|
250
|
+
## License
|
|
251
|
+
|
|
252
|
+
This project is licensed under the MIT License — see the [LICENSE](https://github.com/mcarbonell/dirichlet-regularization/blob/main/LICENSE) file for details.
|