tabdgp 0.1.0__tar.gz → 0.2.0__tar.gz

{tabdgp-0.1.0 → tabdgp-0.2.0}/.github/workflows/release.yml

@@ -15,6 +15,16 @@ jobs:
     steps:
       - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v10.2.0  # setup-uv publishes no major tag
+      # PyPI never takes a version twice, so a tag that names another version stops here
+      - if: github.event_name == 'release'
+        env:
+          TAG: ${{ github.event.release.tag_name }}
+        run: |
+          expected="v$(uv version --short)"
+          if [ "$TAG" != "$expected" ]; then
+            echo "::error::the tag $TAG does not name the package version $expected"
+            exit 1
+          fi
       - run: uv build
       - uses: actions/upload-artifact@v7
         with:

{tabdgp-0.1.0 → tabdgp-0.2.0}/.gitignore

@@ -9,10 +9,15 @@ dist/
 .ipynb_checkpoints/
 .DS_Store
 .claude/
-# local agent instructions (docs/conventions.md is the public part)
+# local agent instructions (docs/conventions.md is the public part) and the roadmap
 CLAUDE.md
+ROADMAP.md
+next.md
 # design notes kept local: the lab notes and the prior-art survey
 lab/
 docs/prior-art.md
 docs/scamd-audit.md
 docs/related-simulators.md
+docs/frontier-priors.md
+# the model reports the survey read, kept local for their licences
+docs/reports/

{tabdgp-0.1.0 → tabdgp-0.2.0}/CITATION.cff

@@ -6,4 +6,6 @@ authors:
     given-names: Alex
 repository-code: https://github.com/adkipnis/tabdgp
 license: MIT
-version: 0.1.0
+version: 0.2.0
+date-released: 2026-10-14
+doi: 10.5281/zenodo.23322845

{tabdgp-0.1.0 → tabdgp-0.2.0}/CONTEXT.md

@@ -29,6 +29,14 @@ of some factor (a school's size, a person's sex). A within-group covariate whose
 between groups is **group-confounded**.
 _Avoid_: level-1 and level-2 variable, cluster-level, contextual variable
 
+**Group share**:
+How much of a within-group node's variance comes from its group part, one value per group of
+one grouping factor, rather than its row part, one value per row: the strength of its group
+confounding. The between-group share measured on its values can be larger, since its row part
+may read group-confounded parents.
+_Avoid_: ICC unqualified (the fitter's, on the outcome), between-group share (what the
+measurement pass records), random-intercept variance
+
 **Quirk**:
 A property of real tables that an independent and identically distributed (i.i.d.) simulator
 lacks and on which an amortized posterior goes wrong: group confounding, empty cells, rare
@@ -41,7 +49,30 @@ _Avoid_: edge case, noise, messiness
 One variable of a dataset's structural causal model (SCM): a grouping factor, a group-level or
 within-group covariate, time, treatment, or the outcome node. A node has a parent set and a
 mechanism; a node without parents is a root.
-_Avoid_: feature, variable unqualified, unit (a neuron of an MLP)
+_Avoid_: feature, variable unqualified, neuron (of an MLP)
+
+**Grouping factor**:
+A root node whose values assign every row to one of its G groups, in metabeta's words "a
+categorical variable whose levels index random effects"; a dataset has none (a flat table),
+one or several. Two grouping factors are **crossed** when their groups combine freely and
+**nested** when each group of the subordinate one lies within one group of the superior one,
+which has fewer groups.
+_Avoid_: factor alone (a categorical column), cluster, subject (one kind of grouping factor),
+random-effects term
+
+**Superior** / **subordinate**:
+A grouping factor is superior to one nested in it, whose groups lie within its own; the nested
+one is subordinate. The row is subordinate to every factor.
+_Avoid_: coarser and finer, parent and child (the graph's edges), higher and lower level
+
+**Unit**:
+What a node is drawn once per: the row, or one grouping factor, in which case the node takes
+one value per group of that factor, repeated across the group's rows (broadcast). A column at
+the row unit is a within-group covariate; one at a grouping factor's unit is group-level. A
+nested grouping factor is a unit subordinate to the one it nests in; crossed ones are units
+without an order between them.
+_Avoid_: level (a factor level, or the multilevel literature's level 1 and 2), hierarchy
+level, grain, scope
 
 **Mechanism**:
 The function that computes a node's values from its parents' values and its own noise, drawn
@@ -64,6 +95,11 @@ ordered), levels, role (grouping factor, covariate, time, treatment, outcome), a
 parameters its values were drawn with.
 _Avoid_: feature, variable unqualified
 
+**Factor level**:
+One of the K categories of a binary, categorical or ordered column, or one group of a
+grouping factor; R's `levels()`.
+_Avoid_: level unqualified, category, class
+
 **Marginal**:
 The distribution a continuous column's values are mapped onto after the SCM, from a catalogue
 of families (skewed, heavy-tailed, bounded, zero-inflated), keeping their ranks.
@@ -95,13 +131,21 @@ _Avoid_: batch, table (the columns alone), sample, corpus (the consumer's set of
 
 **Hyperprior**:
 The one config of distributions from which every quantity of a dataset is drawn: sizes,
-graph, mechanisms, marginals, types, quirks. Its defaults are to be matched to mixedbench's
-registry and then widened; they are what the proofs certify.
+graph, mechanisms, marginals, types, quirks, reached through a preset. The grouped preset's
+defaults are to be matched to mixedbench's registry and then widened; they are what the
+proofs certify.
 _Avoid_: prior (the fitter's, over parameters), config unqualified, meta-distribution
 
+**Preset**:
+A named hyperprior for one kind of table, grouped or flat today, item response, survival or
+causal later: a choice of fields, never a new mechanism. Every draw starts from one; metabeta
+draws from the grouped preset.
+_Avoid_: dataset type, kind, mode
+
 **Override**:
-A point value or a distribution passed for one field of the hyperprior, replacing its default
-for that call. A standalone user passes none; metabeta passes its sizes.
+A point value or a distribution passed for one field of a preset, replacing its default for
+that call. Neither a standalone user nor metabeta passes one by default; an override serves a
+study, an ablation or a size outside the preset.
 _Avoid_: setting, parameter, hyperparameter
 
 **Quirk axis**:
@@ -110,11 +154,18 @@ mixedbench's measurement pass shows that real tables have the quirk strongly eno
 _Avoid_: noise, perturbation, augmentation
 
 **Outcome node**:
-An optional node whose parents are drawn from the covariates, with a family, a link and terms
-a GLMM lacks, that builds white-box misspecified datasets. Just another node; never drawn
-unless asked for.
+An optional node whose parents are drawn from the covariates, with a family, a link, the
+random effects per grouping factor and terms a GLMM lacks, that builds white-box
+misspecified datasets. Just another node; never drawn unless asked for.
 _Avoid_: y, target, label, response unqualified
 
+**Random effect**:
+A per-group parameter of the outcome, an intercept or a slope, drawn jointly per group of a
+grouping factor in metabeta's parameterisation; metabeta's term. Only the outcome node draws
+random effects; a group-level node, latent or observed, is covariate structure, which a GLMM
+takes as given.
+_Avoid_: group effect, random intercept for a latent group-level node
+
 ### The proofs
 
 **Two-sample test**:

{tabdgp-0.1.0 → tabdgp-0.2.0}/PKG-INFO

@@ -1,10 +1,9 @@
 Metadata-Version: 2.5
 Name: tabdgp
-Version: 0.1.0
+Version: 0.2.0
 Summary: A data-generating process for grouped and flat tables, to be matched to a registry of real tables
 Project-URL: Repository, https://github.com/adkipnis/tabdgp
 Project-URL: Issues, https://github.com/adkipnis/tabdgp/issues
-Project-URL: Roadmap, https://github.com/adkipnis/tabdgp/blob/main/ROADMAP.md
 Author-email: Alex Kipnis <adkipnis@users.noreply.github.com>
 License-Expression: MIT
 License-File: LICENSE
@@ -24,6 +23,7 @@ Description-Content-Type: text/markdown
 [![PyPI](https://img.shields.io/pypi/v/tabdgp)](https://pypi.org/project/tabdgp/)
 [![Python](https://img.shields.io/pypi/pyversions/tabdgp)](https://github.com/adkipnis/tabdgp/blob/main/pyproject.toml)
 [![Licence](https://img.shields.io/badge/licence-MIT-blue)](https://github.com/adkipnis/tabdgp/blob/main/LICENSE)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23322845.svg)](https://doi.org/10.5281/zenodo.23322845)
 
 # TabDGP
 
@@ -33,12 +33,14 @@ TabDGP simulates tabular datasets from a random structural causal model (SCM) an
 reproduce the quirks of real data. Each table comes with the graph and the settings that
 generated it.
 
-> **Alpha.** Release 0.1 draws flat tables from defaults that are round guesses; the rest follows
-> the [roadmap](https://github.com/adkipnis/tabdgp/blob/main/ROADMAP.md). Before 1.0, a release
-> may change the dataset a key draws.
+> **Alpha.** Release 0.2 draws grouped and flat tables from defaults that are round guesses;
+> the rest is planned below. Before 1.0, a release may change the dataset a key draws.
 
 ## Highlights
 
+- **Grouped or flat.** One to three grouping factors by default, nested or crossed, with 2 to
+  400 groups each. A covariate varies by row or is constant within the groups of one factor.
+  `Hyperprior.flat()` draws none.
 - **White-box.** Each dataset carries its graph, latent nodes, column metadata and hyperparameters.
 - **Reproducible.** A dataset is a pure function of its key (a seed and an index) and the
   hyperprior, whatever batch or worker draws it.
@@ -55,33 +57,37 @@ pip install tabdgp
 ```python
 from tabdgp import Hyperprior, draw
 
-hyperprior = Hyperprior(rows=100, design_columns=5)  # two overrides, the rest keep their defaults
+hyperprior = Hyperprior.grouped(rows=100, design_columns=5)  # two overrides on the grouped preset
 dataset = draw(seed=0, index=0, hyperprior=hyperprior)  # the key: a seed and an index
+unit = dict(zip(dataset.graph.names, dataset.graph.unit))  # per row, or per group of a factor
 for name, column in dataset.metadata.items():
-    print(name, column.type, dataset.columns[name][:3].round(2))
+    print(name, column.type, unit[name], dataset.columns[name][:3].round(2))
 print(len(dataset.latent), 'latent nodes,', dataset.graph.parents.sum(), 'edges')
 ```
 
 ```text
-x0 binary [1 1 1]
-x1 binary [0 1 0]
-x2 categorical [0 0 0]
-x3 continuous [0.25 0.42 0.1 ]
-7 latent nodes, 8 edges
+g0 categorical g0 [4 5 2]
+g1 categorical g1 [ 6 22 29]
+g2 categorical g2 [7 7 5]
+x0 binary g0 [1 1 1]
+x1 binary row [1 1 1]
+x2 continuous row [0.36 0.69 0.15]
+x3 binary row [0 0 0]
+x4 binary row [0 1 0]
+7 latent nodes, 16 edges
 ```
 
 ## Planned
 
-- **Grouping factors**, with group-level and within-group covariates, time and treatment.
 - **Quirk axes:** group confounding, rare levels, ties, bounds, near dependence and more.
-- **Matched defaults**, fitted to [mixedbench](https://github.com/adkipnis/mixedbench)'s real
-  grouped tables, then widened.
-- **An outcome node**, optional, for white-box misspecified datasets.
+- **Matched defaults**, fitted to and checked against
+  [mixedbench](https://github.com/adkipnis/mixedbench)'s real grouped tables, then widened.
+- **An outcome node**, optional, for white-box misspecified datasets, with within-group time and
+  a binary treatment.
 
 ## Learn more
 
-[`CONTEXT.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTEXT.md) defines the vocabulary,
-and [`ROADMAP.md`](https://github.com/adkipnis/tabdgp/blob/main/ROADMAP.md) gives the roadmap.
+[`CONTEXT.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTEXT.md) defines the vocabulary.
 
 ## Contributing
 

{tabdgp-0.1.0 → tabdgp-0.2.0}/README.md

@@ -2,6 +2,7 @@
 [![PyPI](https://img.shields.io/pypi/v/tabdgp)](https://pypi.org/project/tabdgp/)
 [![Python](https://img.shields.io/pypi/pyversions/tabdgp)](https://github.com/adkipnis/tabdgp/blob/main/pyproject.toml)
 [![Licence](https://img.shields.io/badge/licence-MIT-blue)](https://github.com/adkipnis/tabdgp/blob/main/LICENSE)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.23322845.svg)](https://doi.org/10.5281/zenodo.23322845)
 
 # TabDGP
 
@@ -11,12 +12,14 @@ TabDGP simulates tabular datasets from a random structural causal model (SCM) an
 reproduce the quirks of real data. Each table comes with the graph and the settings that
 generated it.
 
-> **Alpha.** Release 0.1 draws flat tables from defaults that are round guesses; the rest follows
-> the [roadmap](https://github.com/adkipnis/tabdgp/blob/main/ROADMAP.md). Before 1.0, a release
-> may change the dataset a key draws.
+> **Alpha.** Release 0.2 draws grouped and flat tables from defaults that are round guesses;
+> the rest is planned below. Before 1.0, a release may change the dataset a key draws.
 
 ## Highlights
 
+- **Grouped or flat.** One to three grouping factors by default, nested or crossed, with 2 to
+  400 groups each. A covariate varies by row or is constant within the groups of one factor.
+  `Hyperprior.flat()` draws none.
 - **White-box.** Each dataset carries its graph, latent nodes, column metadata and hyperparameters.
 - **Reproducible.** A dataset is a pure function of its key (a seed and an index) and the
   hyperprior, whatever batch or worker draws it.
@@ -33,33 +36,37 @@ pip install tabdgp
 ```python
 from tabdgp import Hyperprior, draw
 
-hyperprior = Hyperprior(rows=100, design_columns=5)  # two overrides, the rest keep their defaults
+hyperprior = Hyperprior.grouped(rows=100, design_columns=5)  # two overrides on the grouped preset
 dataset = draw(seed=0, index=0, hyperprior=hyperprior)  # the key: a seed and an index
+unit = dict(zip(dataset.graph.names, dataset.graph.unit))  # per row, or per group of a factor
 for name, column in dataset.metadata.items():
-    print(name, column.type, dataset.columns[name][:3].round(2))
+    print(name, column.type, unit[name], dataset.columns[name][:3].round(2))
 print(len(dataset.latent), 'latent nodes,', dataset.graph.parents.sum(), 'edges')
 ```
 
 ```text
-x0 binary [1 1 1]
-x1 binary [0 1 0]
-x2 categorical [0 0 0]
-x3 continuous [0.25 0.42 0.1 ]
-7 latent nodes, 8 edges
+g0 categorical g0 [4 5 2]
+g1 categorical g1 [ 6 22 29]
+g2 categorical g2 [7 7 5]
+x0 binary g0 [1 1 1]
+x1 binary row [1 1 1]
+x2 continuous row [0.36 0.69 0.15]
+x3 binary row [0 0 0]
+x4 binary row [0 1 0]
+7 latent nodes, 16 edges
 ```
 
 ## Planned
 
-- **Grouping factors**, with group-level and within-group covariates, time and treatment.
 - **Quirk axes:** group confounding, rare levels, ties, bounds, near dependence and more.
-- **Matched defaults**, fitted to [mixedbench](https://github.com/adkipnis/mixedbench)'s real
-  grouped tables, then widened.
-- **An outcome node**, optional, for white-box misspecified datasets.
+- **Matched defaults**, fitted to and checked against
+  [mixedbench](https://github.com/adkipnis/mixedbench)'s real grouped tables, then widened.
+- **An outcome node**, optional, for white-box misspecified datasets, with within-group time and
+  a binary treatment.
 
 ## Learn more
 
-[`CONTEXT.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTEXT.md) defines the vocabulary,
-and [`ROADMAP.md`](https://github.com/adkipnis/tabdgp/blob/main/ROADMAP.md) gives the roadmap.
+[`CONTEXT.md`](https://github.com/adkipnis/tabdgp/blob/main/CONTEXT.md) defines the vocabulary.
 
 ## Contributing
 

{tabdgp-0.1.0 → tabdgp-0.2.0}/docs/adr/0001-one-dataset-per-draw-keyed-in-numpy.md

@@ -1,7 +1,7 @@
 ---
 status: accepted
 date: 2026-10-11
-amended: 2026-10-11, after the throughput lab note; 2026-10-12, after the column-pooling lab note (#16); 2026-10-12, numpy's version recorded beside TabDGP's
+amended: 2026-10-11, after the throughput lab note; 2026-10-12, after the column-pooling lab note (#16); 2026-10-12, numpy's version recorded beside TabDGP's; 2026-10-14, ADR 0004: metabeta passes no sizes, `draw` takes a preset and has no default
 ---
 
 # 0001 — One dataset per draw, keyed by seed and index, in numpy
@@ -10,7 +10,7 @@ TabDGP's first consumer, `metabeta.mixed` in the private repository metabeta-v3,
 simulated datasets whose sizes, groups and columns all vary; its roadmap puts padding,
 masking, collate and the `.npz` corpus on `metabeta.core`, and its Simulator adds the outcome
 before anything is written. Grilled on
-2026-10-11 (storage, batch layout, tensor library, item 1 of `ROADMAP.md`), the decision is:
+2026-10-11 (storage, batch layout, tensor library, roadmap item 1), the decision is:
 **the seed is the corpus.** Dataset `i` is a pure function of its key `(seed, index)`, the
 hyperprior with its overrides and the installed versions of TabDGP and numpy, whose generator
 promises no stable streams across its releases; the dataset records both. The public call

{tabdgp-0.1.0 → tabdgp-0.2.0}/docs/adr/0002-package-release-first-paper-conditional.md

@@ -59,7 +59,7 @@ a finding the other papers do not carry.**
 
 ## Consequences
 
-- `ROADMAP.md` gains a "Release and paper" section: the release with JOSS, the quirk census
+- The roadmap gains a "Release and paper" section: the release with JOSS, the quirk census
   ownership, the ablations, and the conditional paper.
 - The proofs' summary vector and bar are documented as consistency checks, never claimed as
   realism in the README or a paper.

tabdgp-0.2.0/docs/adr/0003-grouping-factors-as-units-in-the-graph.md

@@ -0,0 +1,121 @@
+---
+status: accepted
+date: 2026-10-14
+amended: 2026-10-14, #28: a column with more levels than its factor has groups stays at the row; #29: the role is spelled `grouping_factor`, a nested pair is superior and subordinate, not coarser and finer, and an override `group_share = 1` is the known exception to every level appearing
+---
+
+# 0003 — Grouping factors are edgeless roots; every node has a unit; confounding is a two-part mechanism
+
+Roadmap item 2 adds grouping to the flat SCM of item 1. Grilled on 2026-10-14 after the
+prior-art search of 2026-10-12 (`docs/frontier-priors.md`, local), the decision is: **a
+grouping factor is a root node without edges whose values assign rows to groups; every other
+node carries a unit, the row or one grouping factor, and is drawn once per element of that
+unit; a row-unit node's group confounding is an explicit share, `group_share`, that mixes a
+group part and a row part.** The words are `CONTEXT.md`'s: grouping factor, unit, factor
+level, group share, random effect.
+
+The witnesses, from mixedbench 1.0.1's registry: 31 % of a documented model's covariates are
+group-level; of 714 tables 56 % have one grouping factor, 20 % crossed, 16 % nested, 8 % both;
+38.8 % of factors are balanced, the rest fitted by a symmetric Dirichlet concentration α with
+p5 0.97 and median 50. The sources of the ideas: PluRel's per-table SCM in topological order
+(group-level nodes before within-group ones), RDB-PFN's parent latent units (latent group-level
+nodes), and Dubey et al. 2026, who found that neither relational prior controls how much of a
+child column its parent explains.
+
+## Decisions
+
+- **The unit is a per-node field**, `unit`, `'row'` or a grouping factor's name, checked at
+  the graph's construction like the latent mask. The grouping factor itself is a root with
+  no edges, its codes its values, the size draw its "noise" (ADR 0004). A node at a grouping
+  factor's unit is drawn on that factor's G group rows: its parents' group values, one noise
+  per group, the mechanism, its z-score and, for a column, its marginal or level codes, all at
+  G rows, then broadcast to the n rows last. Latent values are returned broadcast, so every
+  array of a dataset has length n.
+- **One unit per node.** Nested and crossed factors are both expressed: a node at the
+  subordinate factor of a chain is constant within the superior one too. What is not
+  expressed is a node drawn once per cell of two crossed factors, whose unit would be a set
+  of grouping factors; that is the extension, for item 6's item-response data, and nothing
+  here precludes it.
+- **Edges across units**: a parent is at the child's unit or at a superior one in its nesting
+  chain; no edges between the units of crossed factors; no row-unit parent of a group-level
+  node. A mask on the Cauchy logits; the edge draw is otherwise item 1's. Node order: the
+  grouping factors, then the units superior first (crossed units in a fixed order), the row
+  unit last, latent before observed within a unit.
+- **Which nodes are group-level**: a per-dataset `group_level_share`; each node, latent or
+  observed, draws a grouping factor's unit with that share (the factor uniform among the
+  dataset's), else the row. A column with more levels than its factor has groups stays at
+  the row: it would take one value per group and leave a level no row has, and no real
+  table shows a group-level column with K > G. The known exception is an override
+  `group_share = 1`: it makes a row-unit column constant within its group factor's groups, so
+  one with K > G misses a level. The default never draws 1, and the override asks for a
+  column constant within groups, which cannot show more levels than there are groups. A
+  within-group column with a latent group-level parent is confounded by nothing in the table.
+- **Group confounding is a two-part mechanism.** A row-unit node is
+  `z(sqrt(s) · z(g) + sqrt(1 − s) · z(r))`: g its mechanism over the parents at one grouping
+  factor's unit (and the superior ones in that factor's chain), computed at that unit and
+  broadcast, pure group-level noise when it has no such parent; r its mechanism over every
+  other parent plus its own noise; s its `group_share`, a per-node field. The factor of the
+  group part is drawn uniformly per node. Every z-score runs at the unit of the node being
+  computed, so s is the node's between-group variance share over rows, the statistic the
+  measurement pass records, up to what its row-unit parents carry.
+- **Relations**: each grouping factor after the first nests in the previous one with a
+  per-dataset `nested_share`, else is crossed with every earlier one; crossed assignments are
+  independent. Nested sizes are the size draw applied twice, bottom-up: the subordinate
+  factor's groups over the rows, then the subordinate factor's groups over the superior
+  factor's groups.
+- **The grouping factor as a column**: int64 codes, type `categorical`, role
+  `grouping_factor`, `levels = G`, outside the design-column budget; the role carries its
+  checks (G ≥ 2, no marginal, the size draw's fields as its parameters). The binary /
+  categorical split is about design-matrix coding and applies to covariates only.
+- **Random effects stay the outcome node's.** A group-level node is covariate structure, which
+  a GLMM takes as given; the outcome node (item 5) draws its random effects per grouping
+  factor in metabeta's parameterisation, so that metabeta's model is a special case. Item 5
+  also decides whether a latent group-level node may be a parent of the outcome, an
+  endogeneity that must be a switch.
+- **Time and treatment are deferred** to the item that reads them: a treatment is a binary
+  column with a label nobody in this item reads, and the questions the label raises
+  (positivity, randomised or confounded) are causal-design questions; time's reader is the
+  outcome's random slope.
+
+## Considered options
+
+- **Where the unit lives.** (a) A per-node field, chosen. (b) An edge from the grouping factor
+  to each node at its unit: duplicates the field, and the mechanism must skip a nominal
+  parent. (c) No field, group-level-ness emergent from a lookup mechanism (one random value
+  per factor level) with zero row noise, the random-effects view: cannot promise a constant
+  column, the column module needs the unit anyway, and the group-level share has no knob.
+- **Confounding strength.** (a) Emergent through group-level parents: no knob, and item 9's
+  ablation could not switch it off. (b) A per-node random intercept added with share s:
+  parents' confounding stays uncontrolled, so s is not the node's share. (c) The two-part
+  mechanism, chosen: `group_share = 0` as an override switches confounding off for the whole
+  dataset, by induction over the graph.
+- **Group part per factor** (a simplex of shares per node) over one factor per node: finer,
+  two fields to fit; item 4 may promote it if the pass shows many columns confounded with
+  respect to two factors.
+- **A count of group-level nodes** over a share: the share is the registry's statistic and
+  works whatever the node count.
+- **Dependence between crossed assignments** (RDB-PFN's affinity, a Dirichlet tilt per group
+  of one factor over the other's): a quirk-2 axis without a registry statistic; waits for the
+  measurement pass.
+- **Where a group-level node is z-scored**, G rows or n rows: best practice (Enders and
+  Tofighi 2007 against `scale()` on the long table) is split and binds the fitter, not the
+  generator, since the column's scale is its marginal's. The rule "at the unit of the node
+  being computed" settles it and makes s the row-level share.
+
+## Consequences
+
+- `graph` gains the per-node `unit` and `group_share`, the mask and the order; `mechanism`
+  the arithmetic at a unit, the broadcast and the two parts; `column` the role, the codes of
+  a grouping factor and the marginal at G values; `hyperprior` the fields and the size draw
+  (ADR 0004). Every dataset a key draws changes: a version change.
+- Group confounding is the first quirk axis granted before the measurement pass, since the
+  registry already witnesses it (31 % group-level columns); its default is a round guess,
+  `Beta(0.5, 2)`, fitted in item 4.
+- mixedbench#71 is asked for three statistics: per within-group column × grouping factor the
+  adjusted ICC(1) (the total group share) and the R² on the factor's observed group-level
+  columns (the observed share, after Dubey et al.'s parent gain), whose gap is the latent
+  share; per pair of crossed factors the empty-cell share.
+- Extension points, none built: a latent group-level root "log group size" that group-level
+  nodes may take as a parent (quirk 8, allowed by the edge rule); a set-valued unit for cell
+  effects; per-factor group shares; the crossed-assignment dependence.
+- `mechanism.py`'s "a bias per unit" becomes "per neuron", since unit now has a meaning.

tabdgp-0.2.0/docs/adr/0004-the-sizes-are-the-envelope.md

@@ -0,0 +1,79 @@
+---
+status: accepted
+date: 2026-10-14
+amended: 2026-10-14, #24: a superior factor has fewer groups than the subordinate one, and a point `groups` cannot nest; #28: the floor starts at the lowest value of `groups`, and a range too narrow for the longest chain fails like a point; #29: superior and subordinate, not coarser and finer
+---
+
+# 0004 — The size defaults are metabeta's envelope; presets, not overrides, choose the kind of table
+
+ADR 0001 and root ADR 0005 of metabeta-v3 let `metabeta.mixed` pass the sizes of its envelope
+rules 7 and 8 as overrides. Decided on 2026-10-14, with ADR 0003: all simulation happens in
+TabDGP, and metabeta draws no size of its own. So **rules 7 and 8 are TabDGP's size defaults,
+a count bounded by another field is drawn from its distribution truncated at that field's
+resolved value, and the kind of table is a preset of the hyperprior: `Hyperprior.grouped()`,
+metabeta's corpus, and `Hyperprior.flat()`; `draw(seed, index, hyperprior)` takes a preset
+and has no default.** What the proofs certify is `grouped()`.
+
+## Decisions
+
+- **The group-size draw is rule 7**, since the Gamma(κ) weights of the prior-art search
+  normalise to the symmetric Dirichlet(α) the registry fitted per factor: with probability
+  `balanced_share` the groups are balanced (n / G each, the remainder one more for the first
+  groups); otherwise one row per group, and the remaining n − G rows multinomial over
+  Dirichlet(α) proportions. Two per-factor fields; G non-empty groups always; no retry.
+- **Truncation at resolve.** `rows` resolves first; `groups` draws from its distribution
+  truncated at `rows`, a superior factor's `groups` below the subordinate factor's, and
+  `design_columns` at `rows / 2` (rule 8's one floor, n ≥ 2d, which item 1 did not enforce).
+  A draw clipped afterwards piles mass on the bound: from `LogUniformInt(2, 400)` at
+  n = 20, 57 % of draws would become all-singleton tables.
+- **A superior factor has fewer groups than the subordinate one.** Its `groups` is
+  truncated at one below the subordinate factor's, so a nested pair is never one partition
+  relabelled. Each factor's floor is the lowest value of `groups` (2 by default) plus the
+  number of factors superior to it, so that every superior one still gets that lowest value;
+  a chain of L factors needs at least L + 1 rows by default, and a `rows` override below that
+  fails at resolve. A chain of L factors needs L counts, each below the subordinate one's, so
+  a `groups` with fewer values than the largest `factors` (a point has one) fails at construction
+  unless `factors` stays that small or `nested_share` at 0.
+- **Presets.** `factors` allows 0, and `Hyperprior.flat()` sets it to 0; `grouped()` is the
+  default below. A later kind of table (item response, survival, causal) is a constructor
+  that fixes fields, never a branch in `draw` and never a new mechanism. Neither metabeta nor a
+  standalone user passes an override; an override serves a study, an ablation or a size
+  outside a preset.
+- **The defaults of `grouped()`**, round numbers, the first set agreed in the grill:
+
+  | field | default | source |
+  |---|---|---|
+  | `factors` | `UniformInt(1, 3)` | rule 3 (counts uniform); registry p90 2, max 5; v1 edge 3 |
+  | `groups`, per factor | `LogUniformInt(2, 400)`, truncated at `rows` or below the subordinate factor's | rule 8; registry p50 32, p90 361 |
+  | `balanced_share` | `1/3` | rule 7; registry 0.388 |
+  | `size_alpha`, per factor | `LogUniform(0.5, 500)` | rule 7; registry p5 0.97, median 50 |
+  | `nested_share` | `Uniform(0.2, 0.6)` | a guess; registry 134 nested to 184 crossed among two-factor tables |
+  | `group_level_share` | `Beta(2, 4)` | a guess at the registry's 31 % |
+  | `group_share`, per node | `Beta(0.5, 2)` | a guess, mass near zero |
+
+  The registry's smallest α, 0.025, lies below the floor; item 4's containment test decides
+  whether to widen it.
+
+## Considered options
+
+- **Clip after the draw** over truncation: rejected above.
+- **A superior factor with at most as many groups as the subordinate one**, the first
+  draft: equal counts put one subordinate group in each superior group, the same partition
+  relabelled, two random effects no generalized linear mixed model (GLMM) tells apart.
+  Forced when the subordinate factor has 2 groups, it hit 15.9 % of nested pairs under
+  `grouped()`. Turning such a pair crossed was the other fix: it honours `nested_share` only
+  sometimes.
+- **A kwarg on `draw`** for the kind of table, `flat: bool` or a `Literal` of kinds: a bool
+  is not future-proof, a `Literal` makes every kind a branch inside `draw` that can contradict
+  an override. A preset is the hyperprior itself.
+- **`factors` from 0 by default**, metabeta overriding to 1: the defaults are what metabeta
+  trains on, so the grouped corpus is the preset and the flat table the other preset.
+
+## Consequences
+
+- ADR 0001's "metabeta passes its sizes" is amended: metabeta passes nothing and calls
+  `draw(seed, index, Hyperprior.grouped())`. metabeta's `envelope.md` (rules 7–9) and its
+  `packages/mixed` ADR 0005 say the same on their side.
+- The proofs run on `grouped()`; mixedbench#72's flat sample tests `flat()`.
+- `hyperprior` gains the per-factor fields, the presets and the truncation; a bound names
+  the field it depends on, and `resolveDataset` resolves in dependency order.

{tabdgp-0.1.0 → tabdgp-0.2.0}/docs/conventions.md

@@ -19,11 +19,11 @@ Each module owns one design decision, hidden from the others:
 
 | module | owns | imported by |
 |---|---|---|
-| `hyperprior` | how a field is a point or a distribution, its bounds, its resolution to points, and which fields each column or node draws, by marginal family for a continuous column | every module |
-| `column` | the design-column budget and the post-hoc types: each column's name, type, K and own parameters, what its metadata guarantees, and its values: a sample of its marginal family or its codes, placed by its node's ranks | draw |
-| `graph` | the generating graph: the Cauchy DAG over latent and observed nodes, latent nodes first, the block bonus, each node's family, activation and scales, and what the graph guarantees | mechanism, draw |
-| `mechanism` | how a node's values follow from its parents and its noise: linear and MLP families, relative noise, the z-score | draw |
-| `draw` | the pipeline and its output: the key's generator, threaded through the modules in a fixed order, nodes joined to columns by name, and what one draw returns | the package's `__init__` |
+| `hyperprior` | how a field is a point or a distribution, its bounds, its resolution to points, a count truncated at another field's value, the presets, and which fields each column, node or grouping factor draws, by marginal family for a continuous column | every module |
+| `column` | the design-column budget and the post-hoc types: each column's name, type, K and own parameters, what its metadata guarantees, and its values: a sample of its marginal family or its codes, placed by its node's ranks; the grouping factor as a column, its relation, size draw and codes, and `Grouping`, the unit arithmetic (a unit's chain, the broadcast between units) | graph, mechanism, draw |
+| `graph` | the generating graph: the Cauchy DAG over latent and observed nodes, the block bonus, each node's unit and the mask on edges out of its chain, the order (the grouping factors, then the units superior first, the row last, latent before observed), each node's family, activation and scales, a row-unit node's group share and group factor, and what the graph guarantees | mechanism, draw |
+| `mechanism` | how a node's values follow from its parents and its noise, at a unit: linear and MLP families, the parents read at the unit, relative noise, the z-score at the node's own unit, and a row-unit node's group part and row part mixed by its group share | draw |
+| `draw` | the pipeline and its output: the key's generator, threaded through the modules in a fixed order, the grouping before the graph, nodes joined to columns by name, every column and latent node broadcast to the rows last, and what one draw returns, the grouping factors first | the package's `__init__` |
 
 ## Design
 
@@ -34,6 +34,10 @@ harder to understand or change; it accrues one small edit at a time.
   implementation folds into its caller or callee.
 - **Information hiding.** Split code by what it knows (a node mechanism, a tensor layout, the
   hyperprior's parameterisation), not by the order it runs in.
+- **One home per fact**, in code as in the docs. A rule written out twice (a truncation, the
+  unit order, a loop's initial state) becomes one function, one named value, or a call to the
+  module that owns it. No utils module: a helper two modules need lives in the module that
+  owns its fact.
 - **No pass-throughs.** A method that forwards its arguments unchanged adds nothing; call the
   target directly.
 - **Pull complexity down.** The module absorbs the awkward case so its callers stay simple. A
@@ -60,7 +64,8 @@ harder to understand or change; it accrues one small edit at a time.
 - A module is one lowercase word. A package's `__init__.py` lists its public names in
   `__all__`.
 - `PascalCase` for classes and `Literal` aliases, `camelCase` for functions and methods,
-  `snake_case` for variables, `UPPER_SNAKE_CASE` for constants.
+  `snake_case` for variables and a `Literal`'s values (`'grouping_factor'`), `UPPER_SNAKE_CASE`
+  for constants.
 - A config field that counts is a plural noun (`rows`, `latent_nodes`), without an `n_` prefix.
 - A function in `src/` is named for what it does, a verb and its object (`_resolveColumn`),
   not for what it returns (`_column`). A test helper is read inside an assert, so it may name
@@ -89,6 +94,8 @@ harder to understand or change; it accrues one small edit at a time.
 ### Control flow
 
 - f-strings, one statement per line, a long expression named on the line before.
+- One operation per line: a draw is named before it is cast or transformed. The Bernoulli
+  idiom `rng.random() < share` counts as one draw.
 - A method lists its steps in the order you would describe them; a step with its own `if`
   ladder or loop becomes a method. A search is a plain loop with an explicit check after it,
   not `for`/`else`, `while True` or `next()` on a generator. A loop fills its result as it
@@ -109,9 +116,10 @@ harder to understand or change; it accrues one small edit at a time.
 
 ### Arrays and tensors
 
-- Annotate shapes inline (`# (B, n, p)` for B datasets of n rows and p columns), reshape with
-  `einops` where the library has it, never mutate a reused array in place. Vectorise with
-  broadcasting and advanced indexing, not loops; batch over datasets.
+- Annotate a shape inline on the line that returns it (`# (B, n, p)` for B datasets of n rows
+  and p columns), never in a docstring. Reshape with `einops` where the library has it, never
+  mutate a reused array in place. Vectorise with broadcasting and advanced indexing, not
+  loops; batch over datasets.
 - One letter per quantity, whatever its case: `n` counts rows, in code and in shape comments
   alike, and `d` counts nodes.
 - Every draw takes an explicit generator or key; nothing reads global random state.

{tabdgp-0.1.0 → tabdgp-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "tabdgp"
-version = "0.1.0"
+version = "0.2.0"
 description = "A data-generating process for grouped and flat tables, to be matched to a registry of real tables"
 readme = "README.md"
 license = "MIT"
@@ -24,7 +24,6 @@ dependencies = [
 [project.urls]
 Repository = "https://github.com/adkipnis/tabdgp"
 Issues = "https://github.com/adkipnis/tabdgp/issues"
-Roadmap = "https://github.com/adkipnis/tabdgp/blob/main/ROADMAP.md"
 
 [dependency-groups]
 dev = [