torch-type-nn 0.1.0.dev0__tar.gz

torch_type_nn-0.1.0.dev0/.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+build/
+dist/
+result
+result-*
+.direnv/
+benchmarks/out/
+# benchmark data: pinned in benchmarks/datasets.json, fetched by `nix develop`
+# (or benchmarks/fetch_data.py), never committed
+benchmarks/data/*
+!benchmarks/data/README.md

torch_type_nn-0.1.0.dev0/BOARD.md

@@ -0,0 +1,256 @@
+# Board
+
+Every number on this page is generated by `benchmarks/board.py` from the
+result files it names; the prose only says what each comparison tests.
+Regenerate everything (all cores, CPU) with:
+
+```sh
+nix run .#board-all     # bench.py + scale.py with --jobs $(nproc), then board.py --write BOARD.md
+```
+
+or step by step:
+
+```sh
+nix run .#bench -- all all --seeds 5 --jobs 0 > benchmarks/results/board-per-sample.jsonl
+nix run .#scale -- all --jobs 0                > benchmarks/results/scale-batch32.jsonl
+nix run .#board -- --write BOARD.md
+```
+
+A hold-out gap is **clear** only when it exceeds two standard errors of the
+difference, `2 sqrt(sd_a²/n_a + sd_b²/n_b)` (the C board's rule); otherwise it
+is *within noise*. "params ratio" is the first model's mean parameter count
+over the second's.
+
+## The models
+
+| model | what it is |
+|---|---|
+| **torch type-nn** | the architecture: threshold rule (every decision from back-prop), width on every junction; `TypeNN` + `TypeAdam` |
+| torch type-nn-bic | the same with the BIC prior (`rule="bic"`, measured on training pairs) |
+| torch ref-type-nn | C's `type-nn` exactly: `rule="bic"`, `width_through_depth_probe=False` |
+| torch ref-type-nn-overfit | C's `type-nn-overfit` exactly: threshold rule, `width_through_depth_probe=False` |
+| torch MLP | the C board's baseline: Linear-ReLU-Linear, 8 hidden units (n ≤ 4) or 16, stock Adam |
+| torch MLP, scaled | `ScalableMLP` grown and pruned by the default (threshold) rule, stock Adam |
+| C type-nn, C type-nn-overfit, C c-mlp | hadilq/type-nn `7333bf7`, run on the same machine as its committed reference (deterministic; reproduces digit for digit) |
+
+## Per-sample board
+
+`benchmarks/bench.py`, a line-by-line port of `bench.c`: the same data files
+and parsing, the fixed 70/30 split (xorshift32 Fisher-Yates, seed 34972),
+train-only standardisation, the same per-epoch shuffle, per-sample Adam at
+`lr × 0.1`, epochs and lr per task, 5 initialisation seeds per cell (mean;
+± is the standard deviation over seeds). Only the initialisation RNG differs
+from C (torch's generator), so single runs are not bit-identical to C.
+With `--jobs` every (task, model, seed) runs in its own single-threaded
+process; the results are identical to a serial run, but `train s` is
+measured with all cores busy, so compare it only within one run.
+
+<!-- board:per-sample -->
+Generated by `benchmarks/board.py` from `benchmarks/reference/c-type-nn-7333bf7-30seeds.jsonl`, `benchmarks/results/board-per-sample.jsonl` (5 seeds, batch 1, cpu).
+
+| task | model | hold MSE | ± sd | hold acc | train MSE | params | ± sd | layers | train s | µs/inf |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| xor | C type-nn | n/a | n/a | n/a | 1.67e-02 | 23.6 | 5.8 | 1→1.9 | 0.01 | 0.1 |
+| xor | C type-nn-overfit | n/a | n/a | n/a | 2.03e-04 | 78.8 | 47.4 | 1→2.7 | 0.01 | 0.2 |
+| xor | C c-mlp | n/a | n/a | n/a | 8.76e-03 | 33.0 | 0.0 | 2→2.0 | 0.00 | 0.0 |
+| xor | torch ref-type-nn | n/a | n/a | n/a | 8.40e-06 | 24.8 | 1.8 | 1→2.0 | 67.24 | 319.7 |
+| xor | torch ref-type-nn-overfit | n/a | n/a | n/a | 4.90e-06 | 56.0 | 17.0 | 1→2.4 | 46.32 | 497.0 |
+| xor | **torch type-nn** | n/a | n/a | n/a | 5.11e-06 | 63.0 | 23.7 | 1→2.4 | 46.60 | 1066.2 |
+| xor | torch type-nn-bic | n/a | n/a | n/a | 8.40e-06 | 24.8 | 1.8 | 1→2.0 | 58.89 | 149.1 |
+| xor | torch MLP | n/a | n/a | n/a | 9.61e-29 | 33.0 | 0.0 | 2→2.0 | 12.39 | 66.9 |
+| xor | torch MLP, scaled | n/a | n/a | n/a | 8.34e-02 | 39.0 | 37.1 | 2→3.2 | 23.44 | 97.2 |
+| iris | C type-nn | 0.0248 | 0.0055 | 0.961 | 3.17e-03 | 120.0 | 45.6 | 3→4.0 | 0.38 | 0.6 |
+| iris | C type-nn-overfit | 0.0240 | 0.0053 | 0.961 | 8.43e-04 | 166.2 | 26.7 | 3→4.0 | 0.07 | 0.6 |
+| iris | C c-mlp | 0.0302 | 0.0052 | 0.956 | 1.24e-02 | 67.0 | 0.0 | 2→2.0 | 0.02 | 0.1 |
+| iris | torch ref-type-nn | 0.0331 | 0.0142 | 0.947 | 6.29e-03 | 144.6 | 27.0 | 3→4.0 | 226.46 | 496.2 |
+| iris | torch ref-type-nn-overfit | 0.0272 | 0.0034 | 0.956 | 8.28e-04 | 159.6 | 33.6 | 3→4.0 | 223.01 | 965.7 |
+| iris | **torch type-nn** | 0.0258 | 0.0019 | 0.956 | 6.23e-04 | 180.0 | 15.1 | 3→4.0 | 225.93 | 1105.1 |
+| iris | torch type-nn-bic | 0.0250 | 0.0062 | 0.964 | 1.37e-04 | 134.8 | 41.1 | 3→4.0 | 243.32 | 456.3 |
+| iris | torch MLP | 0.0309 | 0.0043 | 0.956 | 1.48e-02 | 67.0 | 0.0 | 2→2.0 | 41.09 | 65.9 |
+| iris | torch MLP, scaled | 0.0262 | 0.0044 | 0.956 | 6.08e-03 | 71.8 | 16.3 | 2→3.0 | 83.05 | 193.3 |
+| wine | C type-nn | 0.0241 | 0.0070 | 0.956 | 4.21e-04 | 183.2 | 62.2 | 4→5.1 | 0.52 | 0.5 |
+| wine | C type-nn-overfit | 0.0258 | 0.0099 | 0.954 | 2.94e-04 | 378.0 | 47.3 | 4→4.9 | 0.11 | 0.9 |
+| wine | C c-mlp | 0.0243 | 0.0042 | 0.970 | 2.68e-03 | 275.0 | 0.0 | 2→2.0 | 0.03 | 0.2 |
+| wine | torch ref-type-nn | 0.0198 | 0.0070 | 0.959 | 3.20e-05 | 214.6 | 34.6 | 4→5.0 | 249.18 | 589.4 |
+| wine | torch ref-type-nn-overfit | 0.0364 | 0.0107 | 0.933 | 2.72e-05 | 357.0 | 14.6 | 4→5.0 | 239.02 | 1400.0 |
+| wine | **torch type-nn** | 0.0189 | 0.0066 | 0.967 | 6.97e-05 | 486.2 | 15.5 | 4→5.0 | 241.50 | 579.0 |
+| wine | torch type-nn-bic | 0.0264 | 0.0072 | 0.952 | 4.31e-05 | 194.2 | 72.0 | 4→4.8 | 263.42 | 1204.3 |
+| wine | torch MLP | 0.0259 | 0.0032 | 0.956 | 3.87e-03 | 275.0 | 0.0 | 2→2.0 | 37.77 | 71.8 |
+| wine | torch MLP, scaled | 0.0177 | 0.0078 | 0.967 | 3.23e-04 | 124.6 | 32.7 | 2→3.0 | 70.93 | 176.3 |
+| wdbc | C type-nn | 0.0488 | 0.0089 | 0.946 | 2.07e-03 | 153.0 | 36.5 | 3→4.0 | 0.17 | 0.3 |
+| wdbc | C type-nn-overfit | 0.0472 | 0.0077 | 0.948 | 2.85e-03 | 176.6 | 40.5 | 3→3.4 | 0.06 | 0.2 |
+| wdbc | C c-mlp | 0.0459 | 0.0028 | 0.944 | 4.99e-03 | 513.0 | 0.0 | 2→2.0 | 0.09 | 0.3 |
+| wdbc | torch ref-type-nn | 0.0439 | 0.0056 | 0.952 | 4.88e-04 | 175.8 | 36.3 | 3→4.0 | 241.50 | 1055.6 |
+| wdbc | torch ref-type-nn-overfit | 0.0435 | 0.0059 | 0.950 | 5.14e-04 | 184.8 | 55.6 | 3→3.6 | 227.80 | 165.4 |
+| wdbc | **torch type-nn** | 0.0473 | 0.0037 | 0.949 | 4.94e-04 | 220.0 | 40.7 | 3→3.6 | 235.37 | 1032.9 |
+| wdbc | torch type-nn-bic | 0.0436 | 0.0052 | 0.950 | 1.64e-03 | 210.0 | 83.4 | 3→4.0 | 243.94 | 494.9 |
+| wdbc | torch MLP | 0.0471 | 0.0040 | 0.944 | 5.70e-03 | 513.0 | 0.0 | 2→2.0 | 31.36 | 25.5 |
+| wdbc | torch MLP, scaled | 0.0539 | 0.0079 | 0.943 | 1.84e-03 | 132.0 | 23.3 | 2→3.0 | 85.69 | 80.2 |
+| diabetes | C type-nn | 0.0331 | 0.0014 | n/a | 2.72e-02 | 20.1 | 5.5 | 2→2.5 | 0.06 | 0.1 |
+| diabetes | C type-nn-overfit | 0.0374 | 0.0027 | n/a | 2.41e-02 | 33.0 | 9.9 | 2→2.5 | 0.03 | 0.1 |
+| diabetes | C c-mlp | 0.0476 | 0.0038 | n/a | 1.38e-02 | 193.0 | 0.0 | 2→2.0 | 0.04 | 0.2 |
+| diabetes | torch ref-type-nn | 0.0328 | 0.0005 | n/a | 2.73e-02 | 18.6 | 4.9 | 2→2.2 | 263.32 | 547.7 |
+| diabetes | torch ref-type-nn-overfit | 0.0366 | 0.0030 | n/a | 2.43e-02 | 31.8 | 7.8 | 2→2.4 | 264.63 | 247.2 |
+| diabetes | **torch type-nn** | 0.0501 | 0.0145 | n/a | 1.75e-02 | 100.0 | 42.6 | 2→3.0 | 311.33 | 367.8 |
+| diabetes | torch type-nn-bic | 0.0330 | 0.0006 | n/a | 2.78e-02 | 18.0 | 2.1 | 2→2.6 | 288.10 | 503.3 |
+| diabetes | torch MLP | 0.0478 | 0.0038 | n/a | 1.47e-02 | 193.0 | 0.0 | 2→2.0 | 71.59 | 33.9 |
+| diabetes | torch MLP, scaled | 0.0410 | 0.0017 | n/a | 2.05e-02 | 61.6 | 10.3 | 2→3.0 | 137.15 | 142.8 |
+| ionosphere | C type-nn | 0.0932 | 0.0166 | 0.895 | 1.22e-02 | 146.4 | 60.1 | 4→4.8 | 0.23 | 0.5 |
+| ionosphere | C type-nn-overfit | 0.0811 | 0.0183 | 0.911 | 2.35e-03 | 336.9 | 78.6 | 4→4.4 | 0.10 | 1.0 |
+| ionosphere | C c-mlp | 0.1412 | 0.0172 | 0.899 | 1.59e-02 | 577.0 | 0.0 | 2→2.0 | 0.07 | 0.4 |
+| ionosphere | torch ref-type-nn | 0.0919 | 0.0212 | 0.900 | 7.40e-03 | 157.2 | 59.2 | 4→5.0 | 275.86 | 612.0 |
+| ionosphere | torch ref-type-nn-overfit | 0.0899 | 0.0199 | 0.898 | 1.61e-03 | 319.6 | 71.2 | 4→4.6 | 263.69 | 570.3 |
+| ionosphere | **torch type-nn** | 0.0854 | 0.0209 | 0.909 | 7.06e-04 | 389.6 | 66.3 | 4→4.8 | 263.67 | 1315.4 |
+| ionosphere | torch type-nn-bic | 0.0837 | 0.0137 | 0.909 | 5.08e-03 | 212.8 | 26.7 | 4→4.8 | 280.34 | 474.0 |
+| ionosphere | torch MLP | 0.1356 | 0.0128 | 0.908 | 1.57e-02 | 577.0 | 0.0 | 2→2.0 | 46.46 | 66.3 |
+| ionosphere | torch MLP, scaled | 0.1149 | 0.0224 | 0.887 | 8.43e-03 | 167.8 | 17.1 | 2→3.0 | 82.35 | 88.1 |
+<!-- /board:per-sample -->
+
+### Is the port faithful?
+
+Each C-exact torch configuration against the C model it ports. Expect
+*within noise* everywhere: the rules are the same, only the initialisation
+RNG differs.
+
+<!-- board:port -->
+| task | comparison | Δ hold MSE | 2se | verdict | params ratio |
+|---|---|---:|---:|---|---:|
+| xor | torch ref-type-nn vs C type-nn | n/a | n/a | train acc 1.00 vs 0.97 (no hold-out) | 1.05× |
+| xor | torch ref-type-nn-overfit vs C type-nn-overfit | n/a | n/a | both fit (no hold-out) | 0.71× |
+| iris | torch ref-type-nn vs C type-nn | +0.0082 | 0.0128 | higher, within noise | 1.20× |
+| iris | torch ref-type-nn-overfit vs C type-nn-overfit | +0.0031 | 0.0036 | higher, within noise | 0.96× |
+| wine | torch ref-type-nn vs C type-nn | -0.0043 | 0.0068 | lower, within noise | 1.17× |
+| wine | torch ref-type-nn-overfit vs C type-nn-overfit | +0.0106 | 0.0103 | higher, **clear** | 0.94× |
+| wdbc | torch ref-type-nn vs C type-nn | -0.0048 | 0.0059 | lower, within noise | 1.15× |
+| wdbc | torch ref-type-nn-overfit vs C type-nn-overfit | -0.0036 | 0.0059 | lower, within noise | 1.05× |
+| diabetes | torch ref-type-nn vs C type-nn | -0.0003 | 0.0007 | lower, within noise | 0.93× |
+| diabetes | torch ref-type-nn-overfit vs C type-nn-overfit | -0.0008 | 0.0028 | lower, within noise | 0.96× |
+| ionosphere | torch ref-type-nn vs C type-nn | -0.0014 | 0.0199 | lower, within noise | 1.07× |
+| ionosphere | torch ref-type-nn-overfit vs C type-nn-overfit | +0.0089 | 0.0190 | higher, within noise | 0.95× |
+<!-- /board:port -->
+
+### Like for like: one harness, one framework
+
+The comparisons that answer "does the architecture help?". Both sides run in
+the same Python harness with the same data, protocol and seeds; only the
+model differs. (The pre-audit board compared torch type-nn with C's MLP
+instead, which mixes the framework and RNG into the comparison.)
+
+<!-- board:like-for-like -->
+| task | comparison | Δ hold MSE | 2se | verdict | params ratio |
+|---|---|---:|---:|---|---:|
+| xor | **torch type-nn** vs torch MLP | n/a | n/a | both fit (no hold-out) | 1.91× |
+| xor | torch type-nn-bic vs torch MLP | n/a | n/a | both fit (no hold-out) | 0.75× |
+| xor | **torch type-nn** vs torch type-nn-bic | n/a | n/a | both fit (no hold-out) | 2.54× |
+| xor | torch MLP, scaled vs torch MLP | n/a | n/a | train acc 0.90 vs 1.00 (no hold-out) | 1.18× |
+| iris | **torch type-nn** vs torch MLP | -0.0051 | 0.0042 | lower, **clear** | 2.69× |
+| iris | torch type-nn-bic vs torch MLP | -0.0059 | 0.0067 | lower, within noise | 2.01× |
+| iris | **torch type-nn** vs torch type-nn-bic | +0.0008 | 0.0058 | higher, within noise | 1.34× |
+| iris | torch MLP, scaled vs torch MLP | -0.0047 | 0.0055 | lower, within noise | 1.07× |
+| wine | **torch type-nn** vs torch MLP | -0.0070 | 0.0066 | lower, **clear** | 1.77× |
+| wine | torch type-nn-bic vs torch MLP | +0.0006 | 0.0070 | higher, within noise | 0.71× |
+| wine | **torch type-nn** vs torch type-nn-bic | -0.0075 | 0.0087 | lower, within noise | 2.50× |
+| wine | torch MLP, scaled vs torch MLP | -0.0082 | 0.0075 | lower, **clear** | 0.45× |
+| wdbc | **torch type-nn** vs torch MLP | +0.0001 | 0.0049 | higher, within noise | 0.43× |
+| wdbc | torch type-nn-bic vs torch MLP | -0.0036 | 0.0058 | lower, within noise | 0.41× |
+| wdbc | **torch type-nn** vs torch type-nn-bic | +0.0037 | 0.0057 | higher, within noise | 1.05× |
+| wdbc | torch MLP, scaled vs torch MLP | +0.0067 | 0.0079 | higher, within noise | 0.26× |
+| diabetes | **torch type-nn** vs torch MLP | +0.0023 | 0.0134 | higher, within noise | 0.52× |
+| diabetes | torch type-nn-bic vs torch MLP | -0.0147 | 0.0034 | lower, **clear** | 0.09× |
+| diabetes | **torch type-nn** vs torch type-nn-bic | +0.0171 | 0.0130 | higher, **clear** | 5.56× |
+| diabetes | torch MLP, scaled vs torch MLP | -0.0067 | 0.0037 | lower, **clear** | 0.32× |
+| ionosphere | **torch type-nn** vs torch MLP | -0.0502 | 0.0219 | lower, **clear** | 0.68× |
+| ionosphere | torch type-nn-bic vs torch MLP | -0.0519 | 0.0167 | lower, **clear** | 0.37× |
+| ionosphere | **torch type-nn** vs torch type-nn-bic | +0.0017 | 0.0224 | higher, within noise | 1.83× |
+| ionosphere | torch MLP, scaled vs torch MLP | -0.0208 | 0.0231 | lower, within noise | 0.29× |
+<!-- /board:like-for-like -->
+
+### The C board, for reference
+
+<!-- board:c-board -->
+| task | comparison | Δ hold MSE | 2se | verdict | params ratio |
+|---|---|---:|---:|---|---:|
+| xor | C type-nn vs C c-mlp | n/a | n/a | train acc 0.97 vs 0.99 (no hold-out) | 0.72× |
+| xor | C type-nn-overfit vs C c-mlp | n/a | n/a | train acc 1.00 vs 0.99 (no hold-out) | 2.39× |
+| iris | C type-nn vs C c-mlp | -0.0054 | 0.0028 | lower, **clear** | 1.79× |
+| iris | C type-nn-overfit vs C c-mlp | -0.0061 | 0.0027 | lower, **clear** | 2.48× |
+| wine | C type-nn vs C c-mlp | -0.0002 | 0.0030 | lower, within noise | 0.67× |
+| wine | C type-nn-overfit vs C c-mlp | +0.0015 | 0.0039 | higher, within noise | 1.37× |
+| wdbc | C type-nn vs C c-mlp | +0.0028 | 0.0034 | higher, within noise | 0.30× |
+| wdbc | C type-nn-overfit vs C c-mlp | +0.0012 | 0.0030 | higher, within noise | 0.34× |
+| diabetes | C type-nn vs C c-mlp | -0.0145 | 0.0015 | lower, **clear** | 0.10× |
+| diabetes | C type-nn-overfit vs C c-mlp | -0.0101 | 0.0017 | lower, **clear** | 0.17× |
+| ionosphere | C type-nn vs C c-mlp | -0.0480 | 0.0087 | lower, **clear** | 0.25× |
+| ionosphere | C type-nn-overfit vs C c-mlp | -0.0601 | 0.0092 | lower, **clear** | 0.58× |
+<!-- /board:c-board -->
+
+On the C board both rules are clearly better than the MLP on diabetes and
+ionosphere with fewer parameters, and within noise on iris, wine and wdbc.
+
+## Larger data
+
+`benchmarks/scale.py`, mini-batches of 32, Adam at 0.003 (TypeAdam for
+type-nn, stock Adam otherwise), 5 seeds, the same 70/30 xorshift split.
+
+- **digits**: UCI optical digits as shipped in scikit-learn 1.5.2, 1797 × 64
+  pixels, 10 classes (one-hot MSE), 60 epochs.
+- **friedman**: Friedman #1, the fixed Delve / OpenML 564 "fried" file,
+  40 768 × 10 inputs (5 pure noise), 40 epochs. The target is min-max scaled
+  on the training rows, so with `Var(e) = 1` the best reachable hold-out MSE
+  is `1 / span²`; it is computed from the data and reported as the noise floor.
+
+The type-nn and scaled-MLP models are built by the same constructor as the
+per-sample board (`bench.build`); MLP-16 and MLP-64 are fixed one-hidden-layer
+MLPs.
+
+<!-- board:scale -->
+Generated by `benchmarks/board.py` from `benchmarks/results/scale-batch32.jsonl` (5 seeds, batch 32, cpu).
+
+| task | model | hold MSE | ± sd | × noise floor | hold acc | train MSE | params | layers | train s |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| digits | **type-nn** | 0.00620 | 0.00131 | n/a | 0.963 | 0.00023 | 5509 | 7.0 | 38.0 |
+| digits | type-nn-bic | 0.00696 | 0.00028 | n/a | 0.957 | 0.00014 | 3352 | 7.0 | 202.5 |
+| digits | MLP, scaled | 0.00779 | 0.00053 | n/a | 0.964 | 0.00170 | 1323 | 3.0 | 9.0 |
+| digits | MLP-16 | 0.01110 | 0.00074 | n/a | 0.959 | 0.00490 | 1210 | 2.0 | 3.7 |
+| digits | MLP-64 | 0.01089 | 0.00022 | n/a | 0.974 | 0.00343 | 4810 | 2.0 | 4.7 |
+| friedman | **type-nn** | 0.00660 | 0.00037 | 6.65 | n/a | 0.00666 | 18 | 2.0 | 211.7 |
+| friedman | type-nn-bic | 0.00195 | 0.00089 | 1.97 | n/a | 0.00194 | 80 | 3.0 | 254.0 |
+| friedman | MLP, scaled | 0.01377 | 0.01039 | 13.88 | n/a | 0.01362 | 17 | 2.6 | 93.0 |
+| friedman | MLP-16 | 0.00121 | 0.00013 | 1.22 | n/a | 0.00118 | 193 | 2.0 | 56.3 |
+| friedman | MLP-64 | 0.00114 | 0.00005 | 1.15 | n/a | 0.00110 | 769 | 2.0 | 43.2 |
+
+friedman: noise floor (best reachable hold-out MSE, `1 / span²`) = 0.00099
+
+| task | comparison | Δ hold MSE | 2se | verdict | params ratio |
+|---|---|---:|---:|---|---:|
+| digits | **type-nn** vs MLP-16 | -0.0049 | 0.0013 | lower, **clear** | 4.55× |
+| digits | **type-nn** vs MLP-64 | -0.0047 | 0.0012 | lower, **clear** | 1.15× |
+| digits | type-nn-bic vs MLP-64 | -0.0039 | 0.0003 | lower, **clear** | 0.70× |
+| digits | **type-nn** vs type-nn-bic | -0.0008 | 0.0012 | lower, within noise | 1.64× |
+| digits | MLP, scaled vs MLP-16 | -0.0033 | 0.0008 | lower, **clear** | 1.09× |
+| friedman | **type-nn** vs MLP-16 | +0.0054 | 0.0004 | higher, **clear** | 0.09× |
+| friedman | **type-nn** vs MLP-64 | +0.0055 | 0.0003 | higher, **clear** | 0.02× |
+| friedman | type-nn-bic vs MLP-64 | +0.0008 | 0.0008 | higher, **clear** | 0.10× |
+| friedman | **type-nn** vs type-nn-bic | +0.0046 | 0.0009 | higher, **clear** | 0.22× |
+| friedman | MLP, scaled vs MLP-16 | +0.0126 | 0.0093 | higher, **clear** | 0.09× |
+<!-- /board:scale -->
+
+## Notes on honesty
+
+- **Width on every junction** is the default because the architecture says a
+  width probe grows between any two layers, not because of any result. It
+  used to be an opt-in (`type-nn-through`) introduced after Friedman #1
+  results under-grew, and was then judged on the same Friedman hold-out; that
+  selection effect is why its old "best model on the task" claim is not
+  repeated here. The C width rule (width blocked next to the depth probe)
+  remains available as `width_through_depth_probe=False` and is what the
+  `ref-*` rows use.
+- **The threshold rule is the default** for the same reason: the architecture
+  decides every scaling step from what back-prop did to the dummy weights.
+  `rule="bic"` adds the BIC prior, which re-evaluates training pairs with
+  forward passes; both are on the board.
+- **Pre-audit results** are archived in `benchmarks/results/pre-audit/` with
+  what changed since; they are not comparable and are not read by `board.py`.
+- **Task settings** (epochs, lr) are the C board's for the per-sample board
+  and `scale.py`'s constants for the larger data; they are shared by every
+  model and were not tuned per model.

torch_type_nn-0.1.0.dev0/LICENSE

@@ -0,0 +1,23 @@
+MIT License
+
+Copyright (c) 2026 torch-type-nn contributors
+Portions derived from type-nn, Copyright (c) 2026 Hadi Lashkati Ghouchani
+(https://github.com/hadilq/type-nn), used under the MIT License.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

torch_type_nn-0.1.0.dev0/PKG-INFO

@@ -0,0 +1,302 @@
+Metadata-Version: 2.5
+Name: torch-type-nn
+Version: 0.1.0.dev0
+Summary: type-nn for PyTorch: networks of partition functions (And of Ors) that grow and prune their own width, degree and depth
+Project-URL: Homepage, https://github.com/hadilq/torch-type-nn
+Project-URL: Repository, https://github.com/hadilq/torch-type-nn
+Project-URL: Issues, https://github.com/hadilq/torch-type-nn/issues
+Project-URL: Board, https://github.com/hadilq/torch-type-nn/blob/main/BOARD.md
+Project-URL: C reference, https://github.com/hadilq/type-nn
+Project-URL: Design notes, https://hadilq.com/posts/train-the-knowledge/
+Author: torch-type-nn contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: neural-network,partition-function,pruning,pytorch,structure-learning,type-theory
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: torch>=2.2
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: numpy; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# torch-type-nn
+
+A PyTorch port of [type-nn](https://github.com/hadilq/type-nn): a neural
+network whose layers are partition functions of an And over Ors, and which
+grows and prunes its own width, degree and depth while it trains. The
+design is explained in
+[Train the knowledge](https://hadilq.com/posts/train-the-knowledge/).
+
+> Status: alpha (0.1.0.dev0). Forward, gradients and the per-Or Adam step
+> agree with the C reference to 1e-12, and both C scaling rules have exact
+> ports on the board (see [Validation](#validation)). Deliberate differences
+> from C are listed in [docs/DIFFERENCES.md](https://github.com/hadilq/torch-type-nn/blob/main/docs/DIFFERENCES.md); the audit
+> that led to the current defaults is in [docs/AUDIT.md](https://github.com/hadilq/torch-type-nn/blob/main/docs/AUDIT.md).
+
+## Quick start
+
+```python
+import torch
+from torch_type_nn import TypeNN, fit
+
+model = TypeNN(in_features=4, out_features=3)     # birth: round(ln(1 + n m)) layers
+result = fit(model, X, Y, epochs=250, lr=0.005)   # grows and prunes while it trains
+print(model.structure(), model.num_params(), result.counters)
+```
+
+`fit` reproduces type-nn's per-sample protocol by default
+(`batch_size=1`); pass `batch_size=64` (or more) for the batched, GPU-friendly
+path. There is no width, degree or depth to choose.
+
+For your own training loop, drive the structure with a `StructureScaler`:
+
+```python
+from torch_type_nn import TypeNN, TypeAdam, StructureScaler
+
+model = TypeNN(n, m)
+opt = TypeAdam(model, lr=lr)
+scaler = StructureScaler(model, opt, epochs=E, steps_per_epoch=len(loader))
+scaler.begin()
+for epoch in range(E):
+    for x, t in loader:
+        y = model(x)
+        loss = 0.5 * (y - t).pow(2).mean()
+        opt.zero_grad(); loss.backward(); opt.step()
+        scaler.observe(x, y, t)          # residual statistics (and pairs, for rule="bic")
+    scaler.epoch_end()                   # grow / prune at the boundary
+scaler.end()                             # last prune; no probe survives
+```
+
+## Structure learning
+
+One dummy rule on three axes. A **probe** is an identity that back-prop is
+free to move; a probe that back-prop moved past `theta(T) = lr T^(3/4)`
+(`T` its age in optimizer steps) is promoted, and a fresh probe takes its
+place. All scaling up happens early and all scaling down late.
+
+| axis   | probe (grow, `u < 1/3`)                                           | drop (prune, `u >= 2/3`)                       |
+| ------ | ----------------------------------------------------------------- | ---------------------------------------------- |
+| width (Or)  | the previous layer grows a trained unit; the current layer reads it with a dummy weight of exactly 0 | the coordinate, when its weights are back inside the band (a junction keeps its most-moved one) |
+| degree (And)| each And carries one noisy identity Or (w ~ 0, b ~ 1)       | an Or back inside the band (an And keeps its most-moved Or: of two identity Ors one goes) |
+| depth  | an identity layer in the gap, between *any* two layers, with the largest mean `|dL/dx|` | a layer back to identity within the band (one per boundary; folded) |
+
+(`u = step / total`; the band is `theta_band = lr S^(3/4)`, `S` optimizer
+steps per epoch.) Growth is gated on an unexplained residual
+(`MSE > Var(t) / N`). Depth edits fold the best affine fit between `x` and
+`F(x)` into the next layer, so they are near-exact. A network is born with
+`round(ln(1 + n m))` layers, `n` inputs and `m` outputs, each layer's output
+feeding the next, as dense as an MLP.
+
+**Two rules.** `StructureScaler(..., rule=...)`:
+
+- `"threshold"` (default) is the architecture: every decision comes from
+  what back-prop did to the dummy weights; no training pair is stored or
+  re-run. It is the rule of C's `type-nn-overfit`.
+- `"bic"` adds the Bayesian-information prior of the post: a moved probe is
+  promoted only if it also pays `n ln(MSE_without / MSE_with) > k ln n`,
+  where `MSE_without` is *measured* by resetting the item to its identity
+  and re-running the epoch's training pairs (so the scaler keeps one epoch
+  of `(x, t)` in memory), and pruning ablates items while the criterion
+  `n ln MSE + K ln n` allows. It is the rule of C's `type-nn`.
+
+**Width on every junction.** By default a width probe also grows on a
+junction that crosses the depth probe (carried through the probe layer).
+C blocks width there, which leaves a network born with depth ≤ 2 no width
+site while a depth probe exists; `TypeNNAdapter(model,
+width_through_depth_probe=False)` reproduces C.
+
+Both rules and both width rules are on [BOARD.md](https://github.com/hadilq/torch-type-nn/blob/main/BOARD.md), including exact
+ports of the two C models.
+
+**Any architecture, any optimizer.** The scaler is written against a
+small protocol (`torch_type_nn.protocol.Scalable`): the rule lives in
+`StructureScaler`, and what an item *is* (how to insert an identity, measure
+its distance from it, remove it) comes from an adapter. `TypeNN` is wrapped
+in `TypeNNAdapter` automatically. `ScalableMLP` (`Linear`/ReLU stacks) is
+the second family: `MLPAdapter` grows and prunes its width and depth. See
+[docs/ADAPTERS.md](https://github.com/hadilq/torch-type-nn/blob/main/docs/ADAPTERS.md) for writing an adapter. Stock
+`torch.optim` optimizers work too: structural edits are reported as `Edit`s
+with index maps, `follow_structure` remaps the optimizer state through them,
+and the scaler attaches `keep_invariants` so `a >= 1` holds after every step.
+
+## The layer
+
+```python
+import torch
+from torch_type_nn import AndOr, TypeAdam
+
+layer = AndOr(in_features=8, out_features=4, ors_per_unit=2)
+z = layer(torch.randn(32, 8))          # (32, 4)
+```
+
+Unit k of a layer is one And over its Ors r:
+
+```
+Or_kr = w_kr . x + b_kr                sum type
+A_k   = prod_r Or_kr ^ a_kr            product type, a_kr >= 1
+z_k   = sign(A_k) ln(1 + |A_k|)        partition function
+```
+
+- **Batched and padded.** All Ors of a layer live in one `(m, R, n)`
+  tensor; a unit with fewer than `R` Ors has identity Ors (w = 0, b = 1,
+  a = 1) in its free slots. The identity is exactly 1, so padding never
+  changes the output, and its gradients are masked so no optimizer moves
+  it. This is what gives type-nn batching and GPU support.
+- **Exact backward.** `AndOrFunction` computes the gradients in log
+  space, including the one-sided rule at a factor that is exactly 0
+  (cofactor slope when a = 1, flat when a > 1), where autograd through
+  `log|O|` would give NaN.
+- **Per-Or Adam.** `TypeAdam(model, lr)` gives every Or its own step
+  count, so an Or born late gets the same bias correction as one born at
+  the start, and keeps `a >= 1`. It takes the module, not a parameter list,
+  because structural edits replace the parameter tensors; the Adam moments
+  live in the layer (`adam_*` buffers) so edits move them with the
+  weights. Non-type-nn parameters in the same model get ordinary Adam.
+  Stock `torch.optim` optimizers also work on a layer of fixed structure.
+- **Structural primitives.** `add_or`, `drop_or`, `add_unit`,
+  `drop_unit`, `add_input` (a zero column: function preserved exactly),
+  `drop_input`, `compact`. `load_state_dict` accepts a checkpoint of a
+  different structure.
+
+## Validation
+
+Two independent checks against [hadilq/type-nn](https://github.com/hadilq/type-nn):
+
+- **Number for number.** `tests/test_crosscheck_c.py` compiles the C code
+  (`tests/c/crosscheck.c`), builds the same ragged network in both, and
+  compares the forward pass, dL/dx, every dL/dw, dL/db and dL/da, and the
+  weights after two per-Or Adam steps. Everything agrees to 1e-12.
+  `nix flake check` runs it (the flake pins the C source as an input);
+  locally, `TYPE_NN_SRC=/path/to/type-nn pytest`.
+- **The board.** `benchmarks/bench.py` ports `bench.c` exactly: the same
+  files, the same xorshift split and shuffle (checked row for row), per-sample
+  Adam at `lr x 0.1`, the same epochs and learning rates, and 5 seeds per
+  cell. `ref-type-nn` and `ref-type-nn-overfit` are exact configurations of
+  C's two models; `benchmarks/board.py` compares each with the C board
+  (`benchmarks/reference/`, deterministic, reproduced digit for digit) and
+  writes every table of [BOARD.md](https://github.com/hadilq/torch-type-nn/blob/main/BOARD.md) from the result files.
+
+```sh
+nix run .#board-all      # every result on all cores, then BOARD.md's tables
+```
+
+## Development
+
+Everything is wired through `flake.nix`:
+
+```sh
+nix develop            # python + torch + pytest + build tools (or: direnv allow);
+                       # also copies the pinned datasets to benchmarks/data
+pytest                 # run the tests from the checkout
+nix run .#test         # same, without entering the shell
+nix flake check        # build the package, run pytest inside the build (with the
+                       # pinned datasets), ruff
+nix build              # ./result: the installed package
+nix run .#dist         # sdist + wheel in ./dist, checked by twine
+nix run .#publish -- --repository testpypi   # upload (TestPyPI first)
+```
+
+The benchmark datasets are pinned (URL + SRI hash) in
+`benchmarks/datasets.json` and never committed; see
+[benchmarks/data/README.md](https://github.com/hadilq/torch-type-nn/blob/main/benchmarks/data/README.md).
+
+Without Nix: `pip install -e ".[dev]" && python benchmarks/fetch_data.py && pytest`.
+
+### On a GPU
+
+Every test that takes the `device` fixture also runs on `cuda` when a CUDA
+device is visible; `TNN_REQUIRE_CUDA=1` makes a missing GPU an error rather
+than a skip. One script runs the GPU suite (the whole test-suite with
+`TNN_REQUIRE_CUDA=1`, then the board's training loop for every model on the
+device), in two ways:
+
+```sh
+nix run .#test-cuda              # on the host, now: no Nix configuration needed
+nix build .#cuda-tests -L        # in the build sandbox: needs the `cuda` feature
+nix flake check --impure         # adds checks.cuda when /dev/nvidiactl exists
+TORCH_TYPE_NN_CUDA=1 nix flake check --impure   # force it (=0: leave it out)
+nix develop .#cuda               # torch-bin (CUDA 13.0) shell
+nix run .#bench-cuda -- all all --seeds 5 --device cuda
+```
+
+A plain `nix flake check` is a pure evaluation, which cannot see the host, so
+it never adds the GPU check. With `--impure` it adds `checks.cuda` when a GPU
+is visible (`/dev/nvidiactl`) *and* the Nix daemon enables the `cuda` system
+feature (read from `/etc/nix/nix.conf`, `/etc/nix/nix.custom.conf`,
+`NIX_CONFIG`); a GPU without the feature skips the check with a warning.
+`TORCH_TYPE_NN_CUDA=1` / `=0` forces it on / off.
+
+The sandboxed check declares `requiredSystemFeatures = [ "cuda" ]`, so the
+daemon must advertise the feature and expose the GPU to the build:
+
+**NixOS:**
+
+```nix
+programs.nix-required-mounts = {
+  enable = true;
+  presets.nvidia-gpu.enable = true;   # adds the cuda/gpu/opengl features and
+};                                    # mounts the driver + /dev/nvidia* for them
+# Setting nix.settings.system-features replaces the default list, so keep the
+# features you rely on (nixos-test, benchmark, big-parallel, kvm, ...):
+nix.settings.system-features = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
+```
+
+**Other Linux (Nix daemon):** find the directory holding the driver's
+`libcuda.so.1` (`ldconfig -p | grep libcuda.so.1`; e.g. `/usr/lib/x86_64-linux-gnu`
+on Debian/Ubuntu, `/usr/lib` on Arch), then in `/etc/nix/nix.conf`:
+
+```
+extra-system-features = cuda
+extra-sandbox-paths = /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-uvm-tools? /run/opengl-driver/lib=/usr/lib/x86_64-linux-gnu
+```
+
+and restart the daemon (`sudo systemctl restart nix-daemon`). torch-bin looks
+for the driver in `/run/opengl-driver/lib`; the `target=source` form mounts the
+host's driver directory there inside the sandbox only (a trailing `?` marks a
+path that may be missing). With several GPUs add `/dev/nvidia1`, ... Check with
+`nix build .#cuda-tests -L`.
+
+## Publishing
+
+Releases go to PyPI from GitHub Actions with trusted publishing
+(`.github/workflows/publish.yml`): no API token is stored anywhere.
+
+One-time setup, on PyPI → *Your account* → *Publishing* → *Add a new pending
+publisher* → *GitHub*:
+
+| field | value |
+|---|---|
+| PyPI Project Name | `torch-type-nn` |
+| Owner | `hadilq` |
+| Repository name | `torch-type-nn` |
+| Workflow name | `publish.yml` |
+| Environment name | `pypi` |
+
+For TestPyPI, the same on test.pypi.org (a separate account) with environment
+`testpypi`. GitHub creates both environments on first use; add protection
+rules (e.g. required reviewers) under *Settings → Environments* if wanted.
+
+Release:
+
+```sh
+# 1. set the version in pyproject.toml (e.g. 0.1.0), commit, push
+# 2. rehearse: Actions → publish → Run workflow → testpypi
+# 3. release: the tag must equal the version
+git tag v0.1.0 && git push origin v0.1.0
+```
+
+The workflow builds the sdist and wheel, runs the test-suite against the
+built wheel (CPU torch), and only then uploads. Locally, `nix run .#dist`
+builds and checks the same files; `nix run .#publish` uploads them with
+twine and a token.
+
+## License
+
+MIT. Derived from type-nn by Hadi Lashkati Ghouchani (MIT).