django-data-shape 0.1.1__tar.gz → 0.2.0__tar.gz

{django_data_shape-0.1.1 → django_data_shape-0.2.0}/CHANGELOG.md

@@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.0] — 2026-09-01
+
+### Added
+- `FanOut`, which declares how a foreign key's children spread across their
+  parents: a size distribution, a `childless` share for parents with no children
+  at all, a `null` share for nullable columns, and `placement`.
+- `Zipf`, the heavy-tailed weight distribution fan-out is realistically drawn
+  from. A table where every parent has ten children is not merely tidy -- it is
+  the one shape in which the planner is never wrong, because its `n_distinct`
+  average is the truth.
+- Tables load in dependency order, and a cycle of fan-outs is refused by name.
+
+### The two representation decisions
+- **Fan-out reads the parent's real keys rather than assuming the dense `1..N`
+  range this package assigns.** The case that matters is the hybrid: a project
+  builds its fifty companies with the ORM, where the row count is small and the
+  ORM is the right tool, and asks this package only for the two million orders.
+  Referential integrity then holds by construction, because every key emitted
+  came out of the parent table.
+- **A fan-out is a partition of the child key range, not a per-child draw.**
+  Parent `j` owns rows `[start, end)`. A per-child draw cannot be inverted, and
+  "which children belong to parent T" is what a mirrored collection needs. The
+  childless tail and `placement` both fall out of the partition for free.
+
+### Notes
+- `placement` defaults to `arrival`. Emitting children parent by parent gives a
+  perfectly clustered table that no production system has and that flatters
+  every index scan over the foreign key.
+- A self-referential fan-out is refused: it would read keys from a table still
+  empty at load time. Self-referential trees are their own feature.
+- A relation needs a `FanOut` and a plain column refuses one, in both
+  directions.
+
 ## [0.1.1] — 2026-09-01
 
 ### Added
@@ -75,6 +108,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   into `COPY FROM STDIN`, which psycopg 2 cannot do without materialising them
   first.
 
-[Unreleased]: https://github.com/Artui/django-data-shape/compare/v0.1.1...HEAD
+[Unreleased]: https://github.com/Artui/django-data-shape/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/Artui/django-data-shape/compare/v0.1.1...v0.2.0
 [0.1.1]: https://github.com/Artui/django-data-shape/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/Artui/django-data-shape/compare/v0.0.0...v0.1.0

{django_data_shape-0.1.1 → django_data_shape-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: django-data-shape
-Version: 0.1.1
+Version: 0.2.0
 Summary: A realistically shaped test database from Django models: declare cardinality, skew and fan-out, load by COPY, and make the query planner believe it.
 Project-URL: Homepage, https://github.com/Artui/django-data-shape
 Project-URL: Repository, https://github.com/Artui/django-data-shape
@@ -96,12 +96,28 @@ build(shape)
 past the keys it assigned, and runs `ANALYZE` so the planner can see the shape.
 It raises on any backend that is not PostgreSQL rather than degrading quietly.
 
+## Relations
+
+```python
+Table(
+    Order,
+    rows=2_000_000,
+    # A distribution, not a number: giving every parent ten children is the one
+    # shape in which the planner is never wrong, because its n_distinct average
+    # is then the truth.
+    company=FanOut(Zipf(1.2), childless=0.35),
+    ...
+)
+```
+
+The parents can be rows this package built or rows your own code did -- their
+real keys are read, not assumed, so the ORM can own the small tables while this
+owns the large ones.
+
 ## Status
 
-Early. This release covers single tables. Foreign-key fan-out as a distribution,
-physical placement, per-group invariants and template-database reuse are the
-releases after it; declaring a relation raises today rather than generating ids
-that point at nothing.
+Early. Single tables and the model graph. Derived fields, collections copied
+along a join, per-group invariants and template-database reuse come next.
 
 Full documentation: <https://artui.github.io/django-data-shape/>
 

{django_data_shape-0.1.1 → django_data_shape-0.2.0}/README.md

@@ -56,12 +56,28 @@ build(shape)
 past the keys it assigned, and runs `ANALYZE` so the planner can see the shape.
 It raises on any backend that is not PostgreSQL rather than degrading quietly.
 
+## Relations
+
+```python
+Table(
+    Order,
+    rows=2_000_000,
+    # A distribution, not a number: giving every parent ten children is the one
+    # shape in which the planner is never wrong, because its n_distinct average
+    # is then the truth.
+    company=FanOut(Zipf(1.2), childless=0.35),
+    ...
+)
+```
+
+The parents can be rows this package built or rows your own code did -- their
+real keys are read, not assumed, so the ORM can own the small tables while this
+owns the large ones.
+
 ## Status
 
-Early. This release covers single tables. Foreign-key fan-out as a distribution,
-physical placement, per-group invariants and template-database reuse are the
-releases after it; declaring a relation raises today rather than generating ids
-that point at nothing.
+Early. Single tables and the model graph. Derived fields, collections copied
+along a join, per-group invariants and template-database reuse come next.
 
 Full documentation: <https://artui.github.io/django-data-shape/>
 

{django_data_shape-0.1.1 → django_data_shape-0.2.0}/django_data_shape/__init__.py

@@ -8,6 +8,8 @@ from django_data_shape.distributions.distribution import Distribution
 from django_data_shape.distributions.sequential import Sequential
 from django_data_shape.distributions.skew import Skew
 from django_data_shape.distributions.uniform import Uniform
+from django_data_shape.distributions.zipf import Zipf
+from django_data_shape.fan_out import FanOut
 from django_data_shape.invalid_shape import InvalidShape
 from django_data_shape.shape import Shape
 from django_data_shape.shape_not_empty import ShapeNotEmpty
@@ -21,6 +23,7 @@ __all__ = [
     "BuildResult",
     "Constant",
     "Distribution",
+    "FanOut",
     "InvalidShape",
     "Sequential",
     "Shape",
@@ -29,6 +32,7 @@ __all__ = [
     "Table",
     "TableResult",
     "Uniform",
+    "Zipf",
     "UnsupportedBackend",
     "__version__",
     "build",

{django_data_shape-0.1.1 → django_data_shape-0.2.0}/django_data_shape/build.py

@@ -2,14 +2,19 @@
 
 from __future__ import annotations
 
-from typing import Any
+from typing import Any, cast
 
 from django.core.management.color import no_style
 from django.db import DEFAULT_DB_ALIAS, connections, transaction
+from django.db.models import Model
 
 from django_data_shape.build_result import BuildResult
+from django_data_shape.fan_out import FanOut
+from django_data_shape.fan_out_plan import FanOutPlan
 from django_data_shape.generate_rows import generate_rows
+from django_data_shape.order_tables import order_tables
 from django_data_shape.require_postgres import require_postgres
+from django_data_shape.resolve_fan_out import resolve_fan_out
 from django_data_shape.shape import Shape
 from django_data_shape.shape_not_empty import ShapeNotEmpty
 from django_data_shape.table import Table
@@ -41,15 +46,35 @@ def build(shape: Shape, using: str = DEFAULT_DB_ALIAS) -> BuildResult:
     # -- fix the shape, run it again -- fails on a duplicate key rather than on
     # the original problem.
     with transaction.atomic(using=using):
-        for table in shape.tables:
+        # Parents first. Not because the database insists -- Django's foreign
+        # keys are deferred, so any order commits -- but because a fan-out reads
+        # its parent's real keys, and a table with no rows yet has none.
+        for table in order_tables(shape.tables):
             _require_empty(connection, table)
-            loaded = _load(connection, table, shape.seed)
+            plans = _resolve(connection, table, shape.seed)
+            loaded = _load(connection, table, shape.seed, plans)
             _reset_sequence(connection, table)
             _analyze(connection, table)
             results.append(TableResult(table=table.db_table, rows=loaded))
     return BuildResult(tables=tuple(results))
 
 
+def _resolve(connection: Any, table: Table, seed: int) -> dict[str, FanOutPlan]:
+    """Partition each declared relation over the parent keys that exist."""
+    plans: dict[str, FanOutPlan] = {}
+    for name, field in table.relations():
+        plans[name] = resolve_fan_out(
+            cast("FanOut", table.fields[name]),
+            cast("type[Model]", field.related_model),
+            table.rows,
+            seed,
+            table.db_table,
+            name,
+            connection,
+        )
+    return plans
+
+
 def _require_empty(connection: Any, table: Table) -> None:
     """Refuse to build on top of rows that are already there.
 
@@ -72,7 +97,7 @@ def _require_empty(connection: Any, table: Table) -> None:
 # API reached through Django's cursor wrapper, and it appears on no Django base
 # class, so annotating the real wrapper type would mean asserting the checker
 # out of the way on every line that uses it.
-def _load(connection: Any, table: Table, seed: int) -> int:
+def _load(connection: Any, table: Table, seed: int, plans: dict[str, FanOutPlan]) -> int:
     """Stream generated rows into the table with ``COPY FROM STDIN``.
 
     ``bulk_create`` is the obvious alternative and is roughly an order of
@@ -105,7 +130,7 @@ def _load(connection: Any, table: Table, seed: int) -> int:
         # block never learns it needs a rollback, so the next query inside it
         # fails with "current transaction is aborted" instead of a Django error.
         with connection.wrap_database_errors, cursor.copy(statement) as copy:
-            for row in generate_rows(table, seed):
+            for row in generate_rows(table, seed, plans):
                 copy.write_row(
                     (
                         row[0],

{django_data_shape-0.1.1 → django_data_shape-0.2.0}/django_data_shape/distributions/__init__.py

@@ -6,5 +6,6 @@ from django_data_shape.distributions.distribution import Distribution
 from django_data_shape.distributions.sequential import Sequential
 from django_data_shape.distributions.skew import Skew
 from django_data_shape.distributions.uniform import Uniform
+from django_data_shape.distributions.zipf import Zipf
 
-__all__ = ["Bounded", "Constant", "Distribution", "Sequential", "Skew", "Uniform"]
+__all__ = ["Bounded", "Constant", "Distribution", "Sequential", "Skew", "Uniform", "Zipf"]

django_data_shape-0.2.0/django_data_shape/distributions/zipf.py

@@ -0,0 +1,37 @@
+"""Heavy-tailed weights: a few very large, a long tail of very small."""
+
+from __future__ import annotations
+
+import math
+
+from django_data_shape.invalid_shape import InvalidShape
+
+
+class Zipf:
+    """Positive weights following a power law of exponent ``s``.
+
+    The distribution fan-out is realistically drawn from, and the reason
+    declaring fan-out is worth doing at all. A customer table where every
+    customer has ten orders is not merely tidy, it is the one shape in which the
+    planner is never wrong: its ``n_distinct`` average *is* the truth, so a join
+    estimate cannot miss. Give the head a thousand orders and the tail one, and
+    the same estimate is out by orders of magnitude in both directions -- which
+    is what production looks like and what a test database has to reproduce.
+
+    Inverse transform of a Pareto: ``(1 - draw) ** (-1 / s)``. Larger ``s``
+    means a lighter tail; values near 1 are the classic Zipf regime.
+    """
+
+    def __init__(self, s: float = 1.2) -> None:
+        if not math.isfinite(s) or s <= 0:
+            raise InvalidShape(f"Zipf needs a positive finite exponent, got s={s}.")
+        self._s = s
+
+    def value(self, row: int, draw: float) -> object:
+        # ``1 - draw`` rather than ``draw`` so the singularity sits at the top of
+        # the interval: draw is in [0, 1), so 1 - draw is in (0, 1] and never
+        # zero, which keeps the power finite for every possible draw.
+        return (1.0 - draw) ** (-1.0 / self._s)
+
+    def __repr__(self) -> str:
+        return f"Zipf({self._s!r})"

django_data_shape-0.2.0/django_data_shape/fan_out.py

@@ -0,0 +1,82 @@
+"""How a foreign key's children are spread across their parents."""
+
+from __future__ import annotations
+
+from django_data_shape.distributions.distribution import Distribution
+from django_data_shape.invalid_shape import InvalidShape
+
+PLACEMENTS = ("arrival", "grouped")
+
+
+class FanOut:
+    """A distribution over how many children each parent has.
+
+    Declared as a shape over an **already fixed** child row count, never as a
+    multiplier. Rows times fan-out makes cardinality emergent, and a table whose
+    size is emergent is a table nothing can assert about.
+
+    ``sizes`` supplies a relative weight per parent -- ``Zipf()`` for the
+    realistic heavy tail, ``Uniform(1, 10)`` for something flatter. The weights
+    are normalised, so their scale is irrelevant and only their spread matters.
+
+    ``childless`` is the share of parents with **no** children at all. It is
+    called out separately because it is the case hand-written fixtures always
+    omit and the one that changes what a join does: a parent nobody references
+    is the difference between an inner and an outer join returning the same
+    thing and returning different things.
+
+    ``null`` is the share of children whose foreign key is NULL, which is only
+    meaningful on a nullable column. **It thins the partition uniformly after**
+    it is computed, so ``sizes`` describes the pre-null spread. Stated because
+    the alternative -- partitioning only the non-null children -- would make the
+    declared distribution mean something subtly different from what it says.
+
+    ``placement`` decides where children sit physically, and it is not
+    cosmetic. Emitting them parent by parent gives a perfectly clustered table
+    that no production system has and that flatters every index scan; the
+    default interleaves them the way rows really arrive.
+    """
+
+    def __init__(
+        self,
+        sizes: Distribution,
+        *,
+        childless: float = 0.0,
+        null: float = 0.0,
+        placement: str = "arrival",
+    ) -> None:
+        for name, share in (("childless", childless), ("null", null)):
+            if not 0.0 <= share < 1.0:
+                raise InvalidShape(
+                    f"FanOut {name} is a share of the whole and must be in [0, 1), got {share}."
+                )
+        if placement not in PLACEMENTS:
+            raise InvalidShape(
+                f"FanOut placement must be one of {', '.join(PLACEMENTS)}, got {placement!r}."
+            )
+        self._sizes = sizes
+        self._childless = childless
+        self._null = null
+        self._placement = placement
+
+    @property
+    def sizes(self) -> Distribution:
+        return self._sizes
+
+    @property
+    def childless(self) -> float:
+        return self._childless
+
+    @property
+    def null(self) -> float:
+        return self._null
+
+    @property
+    def placement(self) -> str:
+        return self._placement
+
+    def __repr__(self) -> str:
+        return (
+            f"FanOut({self._sizes!r}, childless={self._childless!r}, "
+            f"null={self._null!r}, placement={self._placement!r})"
+        )

django_data_shape-0.2.0/django_data_shape/fan_out_plan.py

@@ -0,0 +1,75 @@
+"""A fan-out resolved against real parent keys."""
+
+from __future__ import annotations
+
+import bisect
+from math import gcd
+
+from django_data_shape.utils import draw
+
+
+class FanOutPlan:
+    """Which parent owns each child row, decided once and answered in O(1).
+
+    A **partition of the child key range**: parent ``j`` owns rows
+    ``[starts[j], starts[j + 1])``. That representation is the whole point, and
+    it was chosen over the obvious alternative -- drawing a parent per child --
+    because a per-child draw **cannot be inverted**. Asking "which children
+    belong to parent T" is what a mirrored collection needs, and against a draw
+    the only answer is to index every row.
+
+    Two consequences fall out for free. A childless parent is one whose range is
+    empty, so the tail everybody forgets is representable rather than
+    approximated. And physical placement becomes a pure question of the order
+    rows are *emitted* in, entirely separate from which parent owns them -- the
+    same split this design keeps finding between one order and another.
+    """
+
+    def __init__(
+        self,
+        keys: list[int],
+        starts: list[int],
+        rows: int,
+        null_stream: int,
+        null_share: float,
+        interleave: bool,
+    ) -> None:
+        self._keys = keys
+        self._starts = starts
+        self._rows = rows
+        self._null_stream = null_stream
+        self._null_share = null_share
+        self._stride = _stride(rows) if interleave else 1
+
+    def key_for(self, row: int) -> int | None:
+        """The parent key for one child row, or None where the key is null."""
+        if self._null_share and draw(self._null_stream, row) < self._null_share:
+            return None
+        slot = (row * self._stride) % self._rows
+        # bisect_right, not left: a parent with an empty range shares its start
+        # with the next one, and bisect_right steps past every duplicate to the
+        # last parent whose range actually begins at or below the slot. That is
+        # what makes a childless parent unreachable rather than special-cased.
+        return self._keys[bisect.bisect_right(self._starts, slot) - 1]
+
+    def sizes(self) -> list[int]:
+        """How many children each parent ended up with, in parent-key order."""
+        bounds = [*self._starts, self._rows]
+        return [bounds[i + 1] - bounds[i] for i in range(len(self._keys))]
+
+
+def _stride(rows: int) -> int:
+    """A multiplier that walks every slot exactly once, scattering as it goes.
+
+    ``row * stride % rows`` is a bijection precisely when the two are coprime,
+    so children land in an order unrelated to their parent without buffering a
+    permutation or holding any state. Starting near the golden ratio of ``rows``
+    gives the low-discrepancy spread that makes consecutive children come from
+    unrelated parents, which is what "arrival order" means physically.
+    """
+    if rows < 3:
+        return 1
+    candidate = max(2, int(rows * 0.6180339887498949))
+    while gcd(candidate, rows) != 1:
+        candidate += 1
+    return candidate

django_data_shape-0.2.0/django_data_shape/generate_rows.py

@@ -0,0 +1,60 @@
+"""Turning a declared table into the tuples COPY will consume."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterator, Mapping
+from functools import partial
+from typing import Any, cast
+
+from django_data_shape.distributions.distribution import Distribution
+from django_data_shape.fan_out_plan import FanOutPlan
+from django_data_shape.table import Table
+from django_data_shape.utils import draw, field_stream
+
+
+def generate_rows(
+    table: Table, seed: int, plans: Mapping[str, FanOutPlan] | None = None
+) -> Iterator[tuple[Any, ...]]:
+    """Yield one tuple per row: the primary key, then each declared column.
+
+    Rows, not model instances. The ORM is the wrong tool at these counts -- the
+    difference between a load measured in seconds and one measured in minutes --
+    and nothing here needs an instance, because no ``save`` will run and no
+    signal should fire.
+
+    Primary keys are a dense ``1..N`` because this package assigns them, which is
+    what lets a child's foreign key be satisfied without a lookup and what makes
+    a self-referential tree acyclic by construction. It also obliges the caller
+    to reset the sequence afterwards; see ``build``.
+
+    ``plans`` carries the resolved fan-out for each relation column. Resolving
+    happens outside this function because it has to read the parent's real keys
+    out of the database, and keeping the query there leaves generation itself
+    backend-neutral and testable without a connection.
+
+    A generator rather than a list: a million tuples is real memory, psycopg
+    writes them one at a time anyway, and materialising the set would buy
+    nothing but peak RSS.
+    """
+    plans = plans or {}
+    # Each column is reduced to one callable of the row index before the loop
+    # starts. At a million rows the loop body runs a million times per column, so
+    # the branch between a fan-out and a value distribution is worth deciding
+    # once rather than a million times.
+    emit: list[Callable[[int], object]] = []
+    for name, _field in table.columns():
+        plan = plans.get(name)
+        if plan is not None:
+            emit.append(plan.key_for)
+            continue
+        distribution = cast("Distribution", table.fields[name])
+        emit.append(
+            partial(_from_distribution, distribution, field_stream(seed, table.db_table, name))
+        )
+
+    for row in range(table.rows):
+        yield (row + 1, *(produce(row) for produce in emit))
+
+
+def _from_distribution(distribution: Distribution, stream: int, row: int) -> object:
+    return distribution.value(row, draw(stream, row))

django_data_shape-0.2.0/django_data_shape/order_tables.py

@@ -0,0 +1,65 @@
+"""Deciding which table to load first."""
+
+from __future__ import annotations
+
+from typing import cast
+
+from django.db.models import Model
+
+from django_data_shape.fan_out import FanOut
+from django_data_shape.invalid_shape import InvalidShape
+from django_data_shape.table import Table
+
+
+def order_tables(tables: tuple[Table, ...]) -> tuple[Table, ...]:
+    """Parents before children, for the reason that is easy to get wrong.
+
+    Not because the database demands it. Django creates every PostgreSQL foreign
+    key ``DEFERRABLE INITIALLY DEFERRED``, so inside one transaction the checks
+    fire at commit and any order would satisfy them.
+
+    The real reason is that a fan-out **reads the parent's keys**, and a table
+    with no rows yet has none. Ordering is therefore a correctness requirement
+    of this package's own design rather than of the schema -- which is worth
+    stating, because the schema is where a reader would look for it and would
+    find nothing.
+
+    A cycle is refused by name. Two tables that each fan out over the other
+    cannot both be loaded second, and no amount of deferring changes that.
+    """
+    by_model = {table.model: table for table in tables}
+    ordered: list[Table] = []
+    state: dict[Table, int] = {}
+
+    def visit(table: Table, trail: tuple[Table, ...]) -> None:
+        mark = state.get(table, 0)
+        if mark == 2:
+            return
+        if mark == 1:
+            names = " -> ".join(t.model.__name__ for t in (*trail, table))
+            raise InvalidShape(
+                f"These tables fan out over each other in a cycle: {names}. One of them has to "
+                "be loaded first, and a cycle means neither can be."
+            )
+        state[table] = 1
+        for name, declared in table.fields.items():
+            if not isinstance(declared, FanOut):
+                continue
+            # cast, not a guard: related_model is Optional on the generic field
+            # descriptor, but Table has already refused a FanOut on anything that
+            # is not a relation. Branching on None here would add a path no test
+            # could reach, which is how 100% branch coverage stops being
+            # achievable honestly.
+            parent = cast("type[Model]", table.model._meta.get_field(name).related_model)
+            # A parent outside the shape is a table the caller built themselves,
+            # which is the supported hybrid: the ORM for the small tables, this
+            # package for the large ones. Its keys get read at resolve time.
+            declared_parent = by_model.get(parent)
+            if declared_parent is not None and declared_parent is not table:
+                visit(declared_parent, (*trail, table))
+        state[table] = 2
+        ordered.append(table)
+
+    for table in tables:
+        visit(table, ())
+    return tuple(ordered)