tabdgp 0.0.1__tar.gz → 0.2.0__tar.gz

{tabdgp-0.0.1 → tabdgp-0.2.0}/.github/workflows/ci.yml

@@ -1,5 +1,6 @@
-# Format check and the fast tests on three operating systems; the slow proofs run by hand or
-# on the cluster (`uv run pytest -m slow`).
+# Format and lint checks and the fast tests,
+# on three operating systems and on the oldest and newest supported Python;
+# the slow proofs run by hand or on the cluster (`uv run pytest -m slow`).
 name: CI
 
 on:
@@ -13,12 +14,15 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
+        python: ['3.12', '3.14']
+    name: ${{ matrix.os }}, Python ${{ matrix.python }}
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v10.2.0  # setup-uv publishes no major tag
         with:
-          python-version: '3.14'
+          python-version: ${{ matrix.python }}
       - run: uv sync
       - run: uv run ruff format --check .
+      - run: uv run ruff check .
       - run: uv run pytest -rs

tabdgp-0.2.0/.github/workflows/release.yml

@@ -0,0 +1,50 @@
+# The release to PyPI by trusted publishing: a published GitHub release uploads its sdist and
+# wheel to PyPI; a manual run uploads them to TestPyPI instead, as the dry run. Zenodo
+# archives the same release event on its own.
+name: Release
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  # the build runs without the publishing token
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: astral-sh/setup-uv@v10.2.0  # setup-uv publishes no major tag
+      # PyPI never takes a version twice, so a tag that names another version stops here
+      - if: github.event_name == 'release'
+        env:
+          TAG: ${{ github.event.release.tag_name }}
+        run: |
+          expected="v$(uv version --short)"
+          if [ "$TAG" != "$expected" ]; then
+            echo "::error::the tag $TAG does not name the package version $expected"
+            exit 1
+          fi
+      - run: uv build
+      - uses: actions/upload-artifact@v7
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # the OIDC token PyPI exchanges for a short-lived upload token
+    steps:
+      - uses: actions/download-artifact@v8
+        with:
+          name: dist
+          path: dist/
+      - if: github.event_name == 'release'
+        uses: pypa/gh-action-pypi-publish@release/v1
+      - if: github.event_name == 'workflow_dispatch'
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/

tabdgp-0.2.0/.gitignore

@@ -0,0 +1,23 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+*.egg-info/
+.pytest_cache/
+.hypothesis/
+.ruff_cache/
+.ipynb_checkpoints/
+.DS_Store
+.claude/
+# local agent instructions (docs/conventions.md is the public part) and the roadmap
+CLAUDE.md
+ROADMAP.md
+next.md
+# design notes kept local: the lab notes and the prior-art survey
+lab/
+docs/prior-art.md
+docs/scamd-audit.md
+docs/related-simulators.md
+docs/frontier-priors.md
+# the model reports the survey read, kept local for their licences
+docs/reports/

tabdgp-0.2.0/CITATION.cff

@@ -0,0 +1,11 @@
+cff-version: 1.2.0
+message: If you use TabDGP, please cite it as below.
+title: 'TabDGP: a data-generating process for grouped and flat tables'
+authors:
+  - family-names: Kipnis
+    given-names: Alex
+repository-code: https://github.com/adkipnis/tabdgp
+license: MIT
+version: 0.2.0
+date-released: 2026-10-14
+doi: 10.5281/zenodo.23322845

tabdgp-0.2.0/CONTEXT.md

@@ -0,0 +1,179 @@
+# TabDGP
+
+A data-generating process for grouped and flat tables: what is drawn, how, and from which
+distributions. The covariate terms are those of metabeta's generalized linear mixed
+model (GLMM) glossary (`packages/mixed/CONTEXT.md` of metabeta-v3, a private repository),
+restated here in its words and adapted where TabDGP differs; statistical terms (grouping
+factor, family, link) are not redefined.
+
+## Language
+
+### Covariates
+
+**Covariate generator**:
+The process that draws a dataset's grouping structure and its typed covariate columns,
+continuous, binary, categorical or ordered, each within-group or group-level, with the quirks
+of real tables. TabDGP is one; it draws an outcome only as the optional outcome node.
+_Avoid_: predictor simulation, feature matrix, X generator, design-matrix generator
+
+**Design matrix**:
+The coding of the covariates the linear predictor sees: an intercept column, z-scored
+continuous columns, 0/1 binary columns, K − 1 treatment-coded columns per categorical.
+Derived from the covariates, never drawn; the hyperprior's design-column budget counts its
+columns besides the intercept.
+_Avoid_: X for the uncoded covariates, features
+
+**Within-group covariate** / **group-level covariate**:
+A covariate that varies within the groups of a factor, versus one constant within each group
+of some factor (a school's size, a person's sex). A within-group covariate whose mean differs
+between groups is **group-confounded**.
+_Avoid_: level-1 and level-2 variable, cluster-level, contextual variable
+
+**Group share**:
+How much of a within-group node's variance comes from its group part, one value per group of
+one grouping factor, rather than its row part, one value per row: the strength of its group
+confounding. The between-group share measured on its values can be larger, since its row part
+may read group-confounded parents.
+_Avoid_: ICC unqualified (the fitter's, on the outcome), between-group share (what the
+measurement pass records), random-intercept variance
+
+**Quirk**:
+A property of real tables that an independent and identically distributed (i.i.d.) simulator
+lacks and on which an amortized posterior goes wrong: group confounding, empty cells, rare
+levels, ties, bounds, irregular time, near dependence, leverage points.
+_Avoid_: edge case, noise, messiness
+
+### The graph
+
+**Node**:
+One variable of a dataset's structural causal model (SCM): a grouping factor, a group-level or
+within-group covariate, time, treatment, or the outcome node. A node has a parent set and a
+mechanism; a node without parents is a root.
+_Avoid_: feature, variable unqualified, neuron (of an MLP)
+
+**Grouping factor**:
+A root node whose values assign every row to one of its G groups, in metabeta's words "a
+categorical variable whose levels index random effects"; a dataset has none (a flat table),
+one or several. Two grouping factors are **crossed** when their groups combine freely and
+**nested** when each group of the subordinate one lies within one group of the superior one,
+which has fewer groups.
+_Avoid_: factor alone (a categorical column), cluster, subject (one kind of grouping factor),
+random-effects term
+
+**Superior** / **subordinate**:
+A grouping factor is superior to one nested in it, whose groups lie within its own; the nested
+one is subordinate. The row is subordinate to every factor.
+_Avoid_: coarser and finer, parent and child (the graph's edges), higher and lower level
+
+**Unit**:
+What a node is drawn once per: the row, or one grouping factor, in which case the node takes
+one value per group of that factor, repeated across the group's rows (broadcast). A column at
+the row unit is a within-group covariate; one at a grouping factor's unit is group-level. A
+nested grouping factor is a unit subordinate to the one it nests in; crossed ones are units
+without an order between them.
+_Avoid_: level (a factor level, or the multilevel literature's level 1 and 2), hierarchy
+level, grain, scope
+
+**Mechanism**:
+The function that computes a node's values from its parents' values and its own noise, drawn
+per node from a mechanism family (linear, a small MLP, a tree). A root's mechanism is its noise.
+_Avoid_: layer, activation (one part of it), edge function
+
+**Generating graph**:
+The directed acyclic graph of a dataset's nodes and their parent sets, returned with every
+dataset.
+_Avoid_: causal graph unqualified, DAG (one property of it), architecture
+
+**Latent node**:
+A node of the generating graph that is not a column of the table, kept in the output so that
+the dataset stays white-box.
+_Avoid_: hidden unit, confounder (a role a latent node can play)
+
+**Column**:
+A node observed in the table, with its metadata: type (continuous, binary, categorical,
+ordered), levels, role (grouping factor, covariate, time, treatment, outcome), and the
+parameters its values were drawn with.
+_Avoid_: feature, variable unqualified
+
+**Factor level**:
+One of the K categories of a binary, categorical or ordered column, or one group of a
+grouping factor; R's `levels()`.
+_Avoid_: level unqualified, category, class
+
+**Marginal**:
+The distribution a continuous column's values are mapped onto after the SCM, from a catalogue
+of families (skewed, heavy-tailed, bounded, zero-inflated), keeping their ranks.
+_Avoid_: noise distribution (the root's), prior
+
+**Post-hoc type**:
+The column type a continuous node is turned into after the SCM, by thresholds or bins on its
+values: binary, categorical, ordered.
+_Avoid_: dtype, encoding (the design matrix's)
+
+### The draw
+
+**Draw**:
+The one public call: given a key and a hyperprior, one dataset, a pure function of both. Who
+draws many loops over indices.
+_Avoid_: sample (a row or a value), generate, batch
+
+**Key**:
+A seed and an index, the identity of one dataset: the same key, hyperprior and versions of
+TabDGP and numpy give the same dataset bitwise, whatever the batch, worker or shard it is
+drawn in.
+_Avoid_: rng, random state, seed alone
+
+**Dataset**:
+What a draw returns: the columns as named arrays in their natural dtype, their metadata, the
+generating graph, the latent nodes' values, the hyperparameters every field resolved to except
+those each column or node draws for itself, and the key. Never padded, never a file.
+_Avoid_: batch, table (the columns alone), sample, corpus (the consumer's set of keys)
+
+**Hyperprior**:
+The one config of distributions from which every quantity of a dataset is drawn: sizes,
+graph, mechanisms, marginals, types, quirks, reached through a preset. The grouped preset's
+defaults are to be matched to mixedbench's registry and then widened; they are what the
+proofs certify.
+_Avoid_: prior (the fitter's, over parameters), config unqualified, meta-distribution
+
+**Preset**:
+A named hyperprior for one kind of table, grouped or flat today, item response, survival or
+causal later: a choice of fields, never a new mechanism. Every draw starts from one; metabeta
+draws from the grouped preset.
+_Avoid_: dataset type, kind, mode
+
+**Override**:
+A point value or a distribution passed for one field of a preset, replacing its default for
+that call. Neither a standalone user nor metabeta passes one by default; an override serves a
+study, an ablation or a size outside the preset.
+_Avoid_: setting, parameter, hyperparameter
+
+**Quirk axis**:
+One field of the hyperprior that controls one quirk of the catalogue, granted only when
+mixedbench's measurement pass shows that real tables have the quirk strongly enough.
+_Avoid_: noise, perturbation, augmentation
+
+**Outcome node**:
+An optional node whose parents are drawn from the covariates, with a family, a link, the
+random effects per grouping factor and terms a GLMM lacks, that builds white-box
+misspecified datasets. Just another node; never drawn unless asked for.
+_Avoid_: y, target, label, response unqualified
+
+**Random effect**:
+A per-group parameter of the outcome, an intercept or a slope, drawn jointly per group of a
+grouping factor in metabeta's parameterisation; metabeta's term. Only the outcome node draws
+random effects; a group-level node, latent or observed, is covariate structure, which a GLMM
+takes as given.
+_Avoid_: group effect, random intercept for a latent group-level node
+
+### The proofs
+
+**Two-sample test**:
+A classifier two-sample test (C2ST) on the summary vector of generated and registry tables:
+the defaults pass when its accuracy stays below the bar.
+_Avoid_: discriminator, realism score
+
+**Containment test**:
+The test that every registry row's summary vector lies inside the support of the generated
+ones.
+_Avoid_: coverage (the fitter's credible-interval property)

{tabdgp-0.0.1 → tabdgp-0.2.0}/CONTRIBUTING.md

@@ -24,7 +24,7 @@ git clone https://github.com/adkipnis/tabdgp
 cd tabdgp
 uv sync
 uv run pytest               # the fast tests
-uv run pytest -m slow       # the proofs against mixedbench's registry
+uv run pytest -m slow       # the proofs against mixedbench's registry, from roadmap item 3
 ```
 
 ## Find your way
@@ -33,13 +33,11 @@ uv run pytest -m slow       # the proofs against mixedbench's registry
 src/tabdgp/     the package; each module's docstring says what it defines and how
 tests/          mirrors src/, fixtures beside the tests
 docs/adr/       the design decisions and why they were taken
-docs/           the conventions and the proofs' method
-lab/            one notebook per small design choice, indexed in lab/README.md
-examples/       the demo notebook
+docs/           the conventions, and from roadmap item 3 the proofs' method
 ```
 
 [`CONTEXT.md`](CONTEXT.md) defines the words (node, mechanism, hyperprior, override, quirk
-axis, outcome node); the covariate terms are metabeta's GLMM glossary's.
+axis, outcome node) and the covariate terms it shares with metabeta's glossary.
 
 ## Pull requests
 
@@ -52,7 +50,7 @@ Three rules keep the generator trustworthy:
 
 - **The defaults are what the proofs certify.** A change to a default distribution of the
   hyperprior, to a proof's seed or to its bar is discussed in an issue first and recorded in
-  the changelog; downstream training corpora depend on them.
+  the release notes; downstream training corpora depend on them.
 - **Written from scratch.** scamd and TabICL's prior are read for ideas; no code is copied
   from either. A mechanism they inspired says so in a comment.
 - **Nothing from downstream.** TabDGP imports nothing from metabeta or mixedbench; the proofs

tabdgp-0.2.0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.5
+Name: tabdgp
+Version: 0.2.0
+Summary: A data-generating process for grouped and flat tables, to be matched to a registry of real tables
+Project-URL: Repository, https://github.com/adkipnis/tabdgp
+Project-URL: Issues, https://github.com/adkipnis/tabdgp/issues
+Author-email: Alex Kipnis <adkipnis@users.noreply.github.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.12
+Requires-Dist: numpy<3,>=2.3
+Requires-Dist: pydantic<3,>=2.12
+Description-Content-Type: text/markdown
+
+[![CI](https://github.com/adkipnis/tabdgp/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/adkipnis/tabdgp/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/tabdgp)](https://pypi.org/project/tabdgp/)
+[![Python](https://img.shields.io/pypi/pyversions/tabdgp)](https://github.com/adkipnis/tabdgp/blob/main/pyproject.toml)
+[![Licence](https://img.shields.io/badge/licence-MIT-blue)](https://github.com/adkipnis/tabdgp/blob/main/LICENSE)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23322845.svg)](https://doi.org/10.5281/zenodo.23322845)
+
+# TabDGP
+
+A data-generating process (DGP) for grouped and flat tables.
+
+TabDGP simulates tabular datasets from a random structural causal model (SCM) and aims to
+reproduce the quirks of real data. Each table comes with the graph and the settings that
+generated it.
+
+> **Alpha.** Release 0.2 draws grouped and flat tables from defaults that are round guesses;
+> the rest is planned below. Before 1.0, a release may change the dataset a key draws.
+
+## Highlights
+
+- **Grouped or flat.** One to three grouping factors by default, nested or crossed, with 2 to
+  400 groups each. A covariate varies by row or is constant within the groups of one factor.
+  `Hyperprior.flat()` draws none.
+- **White-box.** Each dataset carries its graph, latent nodes, column metadata and hyperparameters.
+- **Reproducible.** A dataset is a pure function of its key (a seed and an index) and the
+  hyperprior, whatever batch or worker draws it.
+- **Overridable.** Replace any default distribution by a point value or another distribution.
+
+## Install
+
+```
+pip install tabdgp
+```
+
+## Quickstart
+
+```python
+from tabdgp import Hyperprior, draw
+
+hyperprior = Hyperprior.grouped(rows=100, design_columns=5)  # two overrides on the grouped preset
+dataset = draw(seed=0, index=0, hyperprior=hyperprior)  # the key: a seed and an index
+unit = dict(zip(dataset.graph.names, dataset.graph.unit))  # per row, or per group of a factor
+for name, column in dataset.metadata.items():
+    print(name, column.type, unit[name], dataset.columns[name][:3].round(2))
+print(len(dataset.latent), 'latent nodes,', dataset.graph.parents.sum(), 'edges')
+```
+
+```text
+g0 categorical g0 [4 5 2]
+g1 categorical g1 [ 6 22 29]
+g2 categorical g2 [7 7 5]
+x0 binary g0 [1 1 1]
+x1 binary row [1 1 1]
+x2 continuous row [0.36 0.69 0.15]
+x3 binary row [0 0 0]
+x4 binary row [0 1 0]
+7 latent nodes, 16 edges
+```
+
+## Planned
+
+- **Quirk axes:** group confounding, rare levels, ties, bounds, near dependence and more.
+- **Matched defaults**, fitted to and checked against
+  [mixedbench](https://github.com/adkipnis/mixedbench)'s real grouped tables, then widened.
+- **An outcome node**, optional, for white-box misspecified datasets, with within-group time and
+  a binary treatment.
+
+## Learn more
+
+[`CONTEXT.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTEXT.md) defines the vocabulary.
+
+## Contributing
+
+Issues and pull requests are welcome; see
+[`CONTRIBUTING.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTRIBUTING.md).
+
+## Citing
+
+Please cite the release you used, from
+[`CITATION.cff`](https://github.com/adkipnis/tabdgp/blob/main/CITATION.cff) or GitHub's "Cite
+this repository".
+
+## Licence
+
+[MIT](https://github.com/adkipnis/tabdgp/blob/main/LICENSE).

tabdgp-0.2.0/README.md

@@ -0,0 +1,84 @@
+[![CI](https://github.com/adkipnis/tabdgp/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/adkipnis/tabdgp/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/tabdgp)](https://pypi.org/project/tabdgp/)
+[![Python](https://img.shields.io/pypi/pyversions/tabdgp)](https://github.com/adkipnis/tabdgp/blob/main/pyproject.toml)
+[![Licence](https://img.shields.io/badge/licence-MIT-blue)](https://github.com/adkipnis/tabdgp/blob/main/LICENSE)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23322845.svg)](https://doi.org/10.5281/zenodo.23322845)
+
+# TabDGP
+
+A data-generating process (DGP) for grouped and flat tables.
+
+TabDGP simulates tabular datasets from a random structural causal model (SCM) and aims to
+reproduce the quirks of real data. Each table comes with the graph and the settings that
+generated it.
+
+> **Alpha.** Release 0.2 draws grouped and flat tables from defaults that are round guesses;
+> the rest is planned below. Before 1.0, a release may change the dataset a key draws.
+
+## Highlights
+
+- **Grouped or flat.** One to three grouping factors by default, nested or crossed, with 2 to
+  400 groups each. A covariate varies by row or is constant within the groups of one factor.
+  `Hyperprior.flat()` draws none.
+- **White-box.** Each dataset carries its graph, latent nodes, column metadata and hyperparameters.
+- **Reproducible.** A dataset is a pure function of its key (a seed and an index) and the
+  hyperprior, whatever batch or worker draws it.
+- **Overridable.** Replace any default distribution by a point value or another distribution.
+
+## Install
+
+```
+pip install tabdgp
+```
+
+## Quickstart
+
+```python
+from tabdgp import Hyperprior, draw
+
+hyperprior = Hyperprior.grouped(rows=100, design_columns=5)  # two overrides on the grouped preset
+dataset = draw(seed=0, index=0, hyperprior=hyperprior)  # the key: a seed and an index
+unit = dict(zip(dataset.graph.names, dataset.graph.unit))  # per row, or per group of a factor
+for name, column in dataset.metadata.items():
+    print(name, column.type, unit[name], dataset.columns[name][:3].round(2))
+print(len(dataset.latent), 'latent nodes,', dataset.graph.parents.sum(), 'edges')
+```
+
+```text
+g0 categorical g0 [4 5 2]
+g1 categorical g1 [ 6 22 29]
+g2 categorical g2 [7 7 5]
+x0 binary g0 [1 1 1]
+x1 binary row [1 1 1]
+x2 continuous row [0.36 0.69 0.15]
+x3 binary row [0 0 0]
+x4 binary row [0 1 0]
+7 latent nodes, 16 edges
+```
+
+## Planned
+
+- **Quirk axes:** group confounding, rare levels, ties, bounds, near dependence and more.
+- **Matched defaults**, fitted to and checked against
+  [mixedbench](https://github.com/adkipnis/mixedbench)'s real grouped tables, then widened.
+- **An outcome node**, optional, for white-box misspecified datasets, with within-group time and
+  a binary treatment.
+
+## Learn more
+
+[`CONTEXT.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTEXT.md) defines the vocabulary.
+
+## Contributing
+
+Issues and pull requests are welcome; see
+[`CONTRIBUTING.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTRIBUTING.md).
+
+## Citing
+
+Please cite the release you used, from
+[`CITATION.cff`](https://github.com/adkipnis/tabdgp/blob/main/CITATION.cff) or GitHub's "Cite
+this repository".
+
+## Licence
+
+[MIT](https://github.com/adkipnis/tabdgp/blob/main/LICENSE).

{tabdgp-0.0.1 → tabdgp-0.2.0}/docs/adr/0001-one-dataset-per-draw-keyed-in-numpy.md

@@ -1,18 +1,21 @@
 ---
 status: accepted
 date: 2026-10-11
+amended: 2026-10-11, after the throughput lab note; 2026-10-12, after the column-pooling lab note (#16); 2026-10-12, numpy's version recorded beside TabDGP's; 2026-10-14, ADR 0004: metabeta passes no sizes, `draw` takes a preset and has no default
 ---
 
 # 0001 — One dataset per draw, keyed by seed and index, in numpy
 
-TabDGP's first consumer, `metabeta.mixed`, trains on simulated datasets whose sizes, groups
-and columns all vary; its roadmap puts padding, masking, collate and the `.npz` corpus on
-`metabeta.core`, and its Simulator adds the outcome before anything is written. Grilled on
-2026-10-11 (storage, batch layout, tensor library, item 1 of `ROADMAP.md`), the decision is:
+TabDGP's first consumer, `metabeta.mixed` in the private repository metabeta-v3, trains on
+simulated datasets whose sizes, groups and columns all vary; its roadmap puts padding,
+masking, collate and the `.npz` corpus on `metabeta.core`, and its Simulator adds the outcome
+before anything is written. Grilled on
+2026-10-11 (storage, batch layout, tensor library, roadmap item 1), the decision is:
 **the seed is the corpus.** Dataset `i` is a pure function of its key `(seed, index)`, the
-hyperprior with its overrides and the installed TabDGP version. The public call draws one
-dataset, `draw(seed, index, hyperprior) -> Dataset`, in numpy, at the dataset's own shapes.
-TabDGP never pads, never batches across datasets and never writes a file.
+hyperprior with its overrides and the installed versions of TabDGP and numpy, whose generator
+promises no stable streams across its releases; the dataset records both. The public call
+draws one dataset, `draw(seed, index, hyperprior) -> Dataset`, in numpy, at the dataset's own
+shapes. TabDGP never pads, never batches across datasets and never writes a file.
 
 ## Considered options
 
@@ -55,7 +58,8 @@ TabDGP never pads, never batches across datasets and never writes a file.
   group indices as floats, and both consumers derive what they need from the metadata anyway.
   `Dataset` holds the columns as named one-dimensional arrays in their natural dtype, the
   column metadata, the generating graph with every node including the latent ones, the latent
-  nodes' values, the hyperparameters every field resolved to, and the key.
+  nodes' values, the hyperparameters every field resolved to, and the key. A field each column
+  or node draws for itself is recorded with that column or in the graph instead (#16).
 
 ## Consequences
 
@@ -66,12 +70,23 @@ TabDGP never pads, never batches across datasets and never writes a file.
 - The end-to-end test draws dataset `i` alone and inside two different index sets and asserts
   bitwise equality; the proofs draw their fixed-seed datasets by index and store nothing.
 - The hyperprior and its resolution share one Pydantic type: an override is a point value or
-  a distribution on a field, a resolved hyperparameter a point value on the same field.
-- Throughput is a bar, not a hope: the spec's first ticket is a lab note timing the core draw
-  (graph, MLP propagation, one rank-matched marginal, one binary threshold) on one core at
-  N in {100, 1k, 10k} and the scaling of a process pool over indices. If the core draw falls
-  under about 500 datasets per second per core at the mean v1 size, a padded torch batch
-  behind the same one-dataset interface is the fallback.
+  a distribution on a field, a resolved hyperparameter a point value on the same field. A
+  field each column or node draws for itself keeps its override there, and the column
+  metadata and the graph record the values drawn (#16).
+- Throughput is a bar, not a hope: a node's CPU workers draw datasets at least as fast as its
+  GPU trains on them. For metabeta (4 to 8 cores and one GPU per node, epochs of 4,192
+  datasets in minibatches of 16 to 64) that is at least 1,000 datasets per second on 4
+  workers at the mean v1 size, about 1 ms per dataset. The unpublished throughput lab note
+  (`lab/2026-10-11-throughput/`) measured about 250 per second per core in float32 on an M3,
+  so numpy holds; the draw is bound by arithmetic, not by the interpreter, so a padded torch
+  batch on CPU buys nothing and one on the node's only GPU takes time from training. The
+  bar was first set at 500 per core, before the consumer's numbers were known.
+- Continuous columns and latent values are float32, the precision the network trains in, so
+  `torch.from_numpy` hands them over unconverted; single precision also runs the
+  activations three to four times faster.
+- When online drawing falls behind, or several models (4 to 16) train at once, a consumer
+  draws a corpus once, stores it and lets every run read it, dividing the cost; storage
+  stays the consumer's, TabDGP still writes no file.
 - A DataFrame is `pandas.DataFrame(dataset.columns)` and needs no dependency here.
 - For `metabeta.mixed`'s slice A1: generate per epoch from `(seed, index)` in dataloader
   workers, pin TabDGP, snapshot validation and test sets, draw an epoch's sizes first (its

tabdgp-0.2.0/docs/adr/0002-package-release-first-paper-conditional.md

@@ -0,0 +1,67 @@
+---
+status: accepted
+date: 2026-10-11
+---
+
+# 0002 — Package release first; a paper only with a finding of its own
+
+Three papers already come out of the project TabDGP serves: mixedbench's (registry census,
+contract, suite, opponent lines; NeurIPS Datasets & Benchmarks 2027), mixedsurvey's (the GLMM
+practice survey) and later metabeta's (amortized inference). Asked on 2026-10-11 whether TabDGP
+earns a fourth once item 5 or 6 is reached, the decision is: **a proper package release and a
+JOSS paper as a citable handle; the validation of the matched hyperprior as a section of the
+metabeta paper; a standalone paper only if the per-quirk ablation and a second consumer give it
+a finding the other papers do not carry.**
+
+## Why item 5 or 6 alone is thin
+
+- At item 5 TabDGP is a simulator plus two self-chosen certificates. The two-sample test and the
+  containment test run on a summary vector we picked, against a bar we set, so they show
+  consistency with the registry on those statistics, not that training on the generator helps.
+- The new idea, a hyperprior for grouped tables measured against mixedbench's real tables
+  instead of hand-tuned like TabICL's or TabPFN's prior, is a component. The evidence that a
+  component matters is downstream: an amortized posterior trained on it against one trained on
+  something simpler, scored on real tables. That evidence lives in metabeta.mixed, not here.
+- A fourth paper from one project reads as salami slicing unless it carries a finding of its own.
+
+## What would make a standalone paper
+
+1. **The downstream ablation.** An amortized posterior, or a prior-data fitted network (PFN),
+   trained on TabDGP, on an i.i.d. baseline prior and on a TabICL-style prior, scored on
+   mixedbench's real tables against its reference posteriors. Needs metabeta.mixed's slice A1.
+2. **Which quirks matter.** Each quirk axis switched off in turn, the downstream effect
+   measured. Tells the field which properties of real tables a prior must reproduce; cheap
+   once the training loop exists, and the result most likely to be cited outside this stack.
+3. **A second consumer.** Item 6's survival extension for metasurv turns metabeta's covariate
+   generator into a prior for tabular amortized inference in general.
+4. **The quirk census** from mixedbench's measurement pass (roadmap item 3): how common group
+   confounding, rare levels, heaping and near dependence are across real grouped tables. The
+   mixedbench paper plans a registry census; which paper owns the quirk statistics is decided
+   before the mixedbench paper is drafted, or the TabDGP paper is left with the ablation alone.
+
+## Considered options
+
+- **A standalone paper at item 5.** Rejected above: no downstream result, a self-set bar.
+- **No paper at all, the package referenced from the metabeta paper.** The cheapest; loses a
+  citable handle for standalone users and the per-quirk result, which belongs to no other
+  paper.
+- **Package release plus JOSS, validation in the metabeta paper, a standalone paper
+  conditional on 1–3.** Chosen.
+
+## Venues
+
+| content | venue |
+|---|---|
+| package with the proofs, no downstream result | JOSS (Journal of Open Source Software): short, a DOI, the software is what is reviewed |
+| plus the ablation and the quirk study | TMLR (Transactions on Machine Learning Research): rolling, judged on correctness |
+| same, with a large per-quirk effect | NeurIPS Datasets & Benchmarks (not the same cycle as mixedbench's) or AISTATS |
+| a test of reception first | the NeurIPS Table Representation Learning workshop |
+
+## Consequences
+
+- The roadmap gains a "Release and paper" section: the release with JOSS, the quirk census
+  ownership, the ablations, and the conditional paper.
+- The proofs' summary vector and bar are documented as consistency checks, never claimed as
+  realism in the README or a paper.
+- The per-quirk ablation is designed so that each quirk axis can be switched off by an
+  override, which item 4's hyperprior already allows; no extra code is planned for it here.