django-data-shape 0.1.0__tar.gz → 0.2.0__tar.gz

django_data_shape-0.2.0/CHANGELOG.md

@@ -0,0 +1,114 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0] — 2026-09-01
+
+### Added
+- `FanOut`, which declares how a foreign key's children spread across their
+  parents: a size distribution, a `childless` share for parents with no children
+  at all, a `null` share for nullable columns, and `placement`.
+- `Zipf`, the heavy-tailed weight distribution fan-out is realistically drawn
+  from. A table where every parent has ten children is not merely tidy -- it is
+  the one shape in which the planner is never wrong, because its `n_distinct`
+  average is the truth.
+- Tables load in dependency order, and a cycle of fan-outs is refused by name.
+
+### The two representation decisions
+- **Fan-out reads the parent's real keys rather than assuming the dense `1..N`
+  range this package assigns.** The case that matters is the hybrid: a project
+  builds its fifty companies with the ORM, where the row count is small and the
+  ORM is the right tool, and asks this package only for the two million orders.
+  Referential integrity then holds by construction, because every key emitted
+  came out of the parent table.
+- **A fan-out is a partition of the child key range, not a per-child draw.**
+  Parent `j` owns rows `[start, end)`. A per-child draw cannot be inverted, and
+  "which children belong to parent T" is what a mirrored collection needs. The
+  childless tail and `placement` both fall out of the partition for free.
+
+### Notes
+- `placement` defaults to `arrival`. Emitting children parent by parent gives a
+  perfectly clustered table that no production system has and that flatters
+  every index scan over the foreign key.
+- A self-referential fan-out is refused: it would read keys from a table still
+  empty at load time. Self-referential trees are their own feature.
+- A relation needs a `FanOut` and a plain column refuses one, in both
+  directions.
+
+## [0.1.1] — 2026-09-01
+
+### Added
+- `Bounded`, an optional second protocol for distributions that can say how many
+  distinct values they produce. `Constant` and `Skew` implement it. It is
+  separate from `Distribution` on purpose: adding the method there would make it
+  required, so a custom distribution written against the single-method protocol
+  would stop satisfying it.
+- A declaration that provably cannot be loaded is now refused at declaration
+  time. A `Constant` on a unique column with more than one row, or a `Skew` with
+  fewer values than rows, is arithmetic rather than a subtle problem, and it used
+  to be discovered by the database partway through a load that had already
+  written most of a table. Only single-column uniqueness is checked; multi-column
+  constraints are satisfiable through combinations across independently declared
+  columns, which is an analysis rather than a comparison.
+
+### Fixed
+- `Uniform` with `places` raised `decimal.InvalidOperation` past 28 significant
+  digits -- Python's default context precision -- from inside the `COPY` loop, on
+  a column such as `numeric(30, 2)` that would have accepted the value. The
+  precision needed is now derived from the declared bounds.
+- `Table` and `Shape` attributes are read-only. Every rule they enforce runs once
+  in `__init__`, so while the attributes were writable a declaration could be
+  edited afterwards into one that would have been refused, with nothing
+  re-checking it.
+
+## [0.1.0] — 2026-09-01
+
+### Added
+- The shape vocabulary: `Shape`, `Table`, and the `Skew`, `Uniform`, `Sequential`
+  and `Constant` distributions, behind a single-method `Distribution` protocol.
+- `build()`, which generates rows, loads them with `COPY FROM STDIN`, moves the
+  identity sequence past the keys it assigned, and runs `ANALYZE`. The order is
+  owned by the library: loading into a table analyzed while empty leaves the
+  planner applying old statistics to a new row count, which is a worse lie than
+  having no statistics at all.
+- `InvalidShape`, raised at declaration time, and `UnsupportedBackend`, raised
+  for any connection that is not PostgreSQL. Generation is backend-neutral;
+  `COPY` and planner statistics are not, and degrading quietly would produce the
+  false confidence this package exists to remove.
+- Primary keys are assigned as a dense `1..N` range rather than declared. That is
+  what will let a foreign key be satisfied without a lookup once relations land,
+  and what makes a self-referential tree acyclic by construction.
+- A field carrying a Django `default=` is filled with the value `save()` would
+  have written, because defaults are applied by `save()` and nothing here calls
+  it. A callable default is refused instead of guessed: `uuid4` varies per row
+  and `dict` does not, and nothing on the field distinguishes them.
+
+- `ShapeNotEmpty`, raised before anything is written when a target table already
+  holds rows. Keys start at 1 on every build, so the collision was previously a
+  unique-violation naming an index, which says nothing about what to do instead.
+- The whole build runs in one transaction, so a shape whose second table fails
+  leaves nothing behind and can be re-run after a fix.
+- Every declared value passes through its field's `get_db_prep_save`. Without it
+  a naive datetime was stored verbatim rather than localised -- hours from where
+  `save()` puts it under a non-UTC `TIME_ZONE` -- and a `JSONField` could not be
+  written at all.
+
+### Notes
+- Relations are refused in both directions: declaring one raises, and so does
+  omitting one that cannot be null. An optional foreign key may be omitted and
+  loads entirely `NULL`, which is documented rather than left to be discovered.
+- Only integer primary keys are supported. Any other kind is refused, rather than
+  a dense `1..N` integer range being written into a character column.
+- psycopg 3 is required and psycopg 2 is refused by name. Rows stream straight
+  into `COPY FROM STDIN`, which psycopg 2 cannot do without materialising them
+  first.
+
+[Unreleased]: https://github.com/Artui/django-data-shape/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/Artui/django-data-shape/compare/v0.1.1...v0.2.0
+[0.1.1]: https://github.com/Artui/django-data-shape/compare/v0.1.0...v0.1.1
+[0.1.0]: https://github.com/Artui/django-data-shape/compare/v0.0.0...v0.1.0

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: django-data-shape
-Version: 0.1.0
+Version: 0.2.0
 Summary: A realistically shaped test database from Django models: declare cardinality, skew and fan-out, load by COPY, and make the query planner believe it.
 Project-URL: Homepage, https://github.com/Artui/django-data-shape
 Project-URL: Repository, https://github.com/Artui/django-data-shape
@@ -96,12 +96,28 @@ build(shape)
 past the keys it assigned, and runs `ANALYZE` so the planner can see the shape.
 It raises on any backend that is not PostgreSQL rather than degrading quietly.
 
+## Relations
+
+```python
+Table(
+    Order,
+    rows=2_000_000,
+    # A distribution, not a number: giving every parent ten children is the one
+    # shape in which the planner is never wrong, because its n_distinct average
+    # is then the truth.
+    company=FanOut(Zipf(1.2), childless=0.35),
+    ...
+)
+```
+
+The parents can be rows this package built or rows your own code did -- their
+real keys are read, not assumed, so the ORM can own the small tables while this
+owns the large ones.
+
 ## Status
 
-Early. This release covers single tables. Foreign-key fan-out as a distribution,
-physical placement, per-group invariants and template-database reuse are the
-releases after it; declaring a relation raises today rather than generating ids
-that point at nothing.
+Early. Single tables and the model graph. Derived fields, collections copied
+along a join, per-group invariants and template-database reuse come next.
 
 Full documentation: <https://artui.github.io/django-data-shape/>
 

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/README.md

@@ -56,12 +56,28 @@ build(shape)
 past the keys it assigned, and runs `ANALYZE` so the planner can see the shape.
 It raises on any backend that is not PostgreSQL rather than degrading quietly.
 
+## Relations
+
+```python
+Table(
+    Order,
+    rows=2_000_000,
+    # A distribution, not a number: giving every parent ten children is the one
+    # shape in which the planner is never wrong, because its n_distinct average
+    # is then the truth.
+    company=FanOut(Zipf(1.2), childless=0.35),
+    ...
+)
+```
+
+The parents can be rows this package built or rows your own code did -- their
+real keys are read, not assumed, so the ORM can own the small tables while this
+owns the large ones.
+
 ## Status
 
-Early. This release covers single tables. Foreign-key fan-out as a distribution,
-physical placement, per-group invariants and template-database reuse are the
-releases after it; declaring a relation raises today rather than generating ids
-that point at nothing.
+Early. Single tables and the model graph. Derived fields, collections copied
+along a join, per-group invariants and template-database reuse come next.
 
 Full documentation: <https://artui.github.io/django-data-shape/>
 

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/django_data_shape/__init__.py

@@ -2,11 +2,14 @@
 
 from django_data_shape.build import build
 from django_data_shape.build_result import BuildResult
+from django_data_shape.distributions.bounded import Bounded
 from django_data_shape.distributions.constant import Constant
 from django_data_shape.distributions.distribution import Distribution
 from django_data_shape.distributions.sequential import Sequential
 from django_data_shape.distributions.skew import Skew
 from django_data_shape.distributions.uniform import Uniform
+from django_data_shape.distributions.zipf import Zipf
+from django_data_shape.fan_out import FanOut
 from django_data_shape.invalid_shape import InvalidShape
 from django_data_shape.shape import Shape
 from django_data_shape.shape_not_empty import ShapeNotEmpty
@@ -16,9 +19,11 @@ from django_data_shape.unsupported_backend import UnsupportedBackend
 from django_data_shape.version import __version__
 
 __all__ = [
+    "Bounded",
     "BuildResult",
     "Constant",
     "Distribution",
+    "FanOut",
     "InvalidShape",
     "Sequential",
     "Shape",
@@ -27,6 +32,7 @@ __all__ = [
     "Table",
     "TableResult",
     "Uniform",
+    "Zipf",
     "UnsupportedBackend",
     "__version__",
     "build",

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/django_data_shape/build.py

@@ -2,14 +2,19 @@
 
 from __future__ import annotations
 
-from typing import Any
+from typing import Any, cast
 
 from django.core.management.color import no_style
 from django.db import DEFAULT_DB_ALIAS, connections, transaction
+from django.db.models import Model
 
 from django_data_shape.build_result import BuildResult
+from django_data_shape.fan_out import FanOut
+from django_data_shape.fan_out_plan import FanOutPlan
 from django_data_shape.generate_rows import generate_rows
+from django_data_shape.order_tables import order_tables
 from django_data_shape.require_postgres import require_postgres
+from django_data_shape.resolve_fan_out import resolve_fan_out
 from django_data_shape.shape import Shape
 from django_data_shape.shape_not_empty import ShapeNotEmpty
 from django_data_shape.table import Table
@@ -41,15 +46,35 @@ def build(shape: Shape, using: str = DEFAULT_DB_ALIAS) -> BuildResult:
     # -- fix the shape, run it again -- fails on a duplicate key rather than on
     # the original problem.
     with transaction.atomic(using=using):
-        for table in shape.tables:
+        # Parents first. Not because the database insists -- Django's foreign
+        # keys are deferred, so any order commits -- but because a fan-out reads
+        # its parent's real keys, and a table with no rows yet has none.
+        for table in order_tables(shape.tables):
             _require_empty(connection, table)
-            loaded = _load(connection, table, shape.seed)
+            plans = _resolve(connection, table, shape.seed)
+            loaded = _load(connection, table, shape.seed, plans)
             _reset_sequence(connection, table)
             _analyze(connection, table)
             results.append(TableResult(table=table.db_table, rows=loaded))
     return BuildResult(tables=tuple(results))
 
 
+def _resolve(connection: Any, table: Table, seed: int) -> dict[str, FanOutPlan]:
+    """Partition each declared relation over the parent keys that exist."""
+    plans: dict[str, FanOutPlan] = {}
+    for name, field in table.relations():
+        plans[name] = resolve_fan_out(
+            cast("FanOut", table.fields[name]),
+            cast("type[Model]", field.related_model),
+            table.rows,
+            seed,
+            table.db_table,
+            name,
+            connection,
+        )
+    return plans
+
+
 def _require_empty(connection: Any, table: Table) -> None:
     """Refuse to build on top of rows that are already there.
 
@@ -72,7 +97,7 @@ def _require_empty(connection: Any, table: Table) -> None:
 # API reached through Django's cursor wrapper, and it appears on no Django base
 # class, so annotating the real wrapper type would mean asserting the checker
 # out of the way on every line that uses it.
-def _load(connection: Any, table: Table, seed: int) -> int:
+def _load(connection: Any, table: Table, seed: int, plans: dict[str, FanOutPlan]) -> int:
     """Stream generated rows into the table with ``COPY FROM STDIN``.
 
     ``bulk_create`` is the obvious alternative and is roughly an order of
@@ -105,7 +130,7 @@ def _load(connection: Any, table: Table, seed: int) -> int:
         # block never learns it needs a rollback, so the next query inside it
         # fails with "current transaction is aborted" instead of a Django error.
         with connection.wrap_database_errors, cursor.copy(statement) as copy:
-            for row in generate_rows(table, seed):
+            for row in generate_rows(table, seed, plans):
                 copy.write_row(
                     (
                         row[0],

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/django_data_shape/distributions/__init__.py

@@ -1,9 +1,11 @@
 """The declared value distributions."""
 
+from django_data_shape.distributions.bounded import Bounded
 from django_data_shape.distributions.constant import Constant
 from django_data_shape.distributions.distribution import Distribution
 from django_data_shape.distributions.sequential import Sequential
 from django_data_shape.distributions.skew import Skew
 from django_data_shape.distributions.uniform import Uniform
+from django_data_shape.distributions.zipf import Zipf
 
-__all__ = ["Constant", "Distribution", "Sequential", "Skew", "Uniform"]
+__all__ = ["Bounded", "Constant", "Distribution", "Sequential", "Skew", "Uniform", "Zipf"]

django_data_shape-0.2.0/django_data_shape/distributions/bounded.py

@@ -0,0 +1,25 @@
+"""Distributions that can only ever produce so many different values."""
+
+from __future__ import annotations
+
+from typing import Protocol, runtime_checkable
+
+
+@runtime_checkable
+class Bounded(Protocol):
+    """A distribution with a known, finite number of distinct values.
+
+    Deliberately a **second** protocol rather than a method on
+    :class:`~django_data_shape.distributions.distribution.Distribution`. Adding
+    it there would make it required, and a custom distribution written against
+    the single-method protocol would stop satisfying it -- so the one thing this
+    exists to prevent, a declaration that cannot describe a database, would be
+    bought by breaking every declaration someone had already written.
+
+    Structural and runtime-checkable, so a distribution opts in by having the
+    method and nothing has to register anywhere. A distribution that cannot
+    answer -- one drawing from a continuous range, say -- simply does not
+    implement it, and is treated as unbounded rather than as suspicious.
+    """
+
+    def distinct_values(self) -> int: ...

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/django_data_shape/distributions/constant.py

@@ -22,5 +22,9 @@ class Constant:
     def value(self, row: int, draw: float) -> object:
         return self._value
 
+    def distinct_values(self) -> int:
+        """One, by definition. See ``Bounded``."""
+        return 1
+
     def __repr__(self) -> str:
         return f"Constant({self._value!r})"

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/django_data_shape/distributions/skew.py

@@ -67,5 +67,9 @@ class Skew:
         # is cheaper than renormalising on every draw.
         return self._values[-1]
 
+    def distinct_values(self) -> int:
+        """How many values were declared. See ``Bounded``."""
+        return len(self._values)
+
     def __repr__(self) -> str:
         return f"Skew({self._weights!r})"

{django_data_shape-0.1.0 → django_data_shape-0.2.0}/django_data_shape/distributions/uniform.py

@@ -3,7 +3,7 @@
 from __future__ import annotations
 
 import math
-from decimal import Decimal
+from decimal import Decimal, localcontext
 
 from django_data_shape.invalid_shape import InvalidShape
 
@@ -38,12 +38,26 @@ class Uniform:
         self._low = low
         self._high = high
         self._places = places
+        # Python's default decimal context carries 28 significant digits, and
+        # rounding past it raises InvalidOperation -- which used to surface from
+        # inside the COPY loop, on a numeric(30, 2) column that would have
+        # accepted the value perfectly well. The precision needed is the integer
+        # digits of the widest bound plus the decimal places, and the guard
+        # covers the rounding step itself.
+        magnitude = max(abs(low), abs(high), 1.0)
+        self._precision = max(28, int(math.log10(magnitude)) + (places or 0) + 5)
 
     def value(self, row: int, draw: float) -> object:
         raw = self._low + draw * (self._high - self._low)
         if self._places is None:
             return raw
-        return round(Decimal(repr(raw)), self._places)
+        with localcontext() as context:
+            context.prec = self._precision
+            # repr() rather than the float itself: Decimal(float) takes the full
+            # binary expansion, which rounds tie cases the other way --
+            # Decimal(2.675) rounds to 2.67 where Decimal(repr(2.675)) gives the
+            # 2.68 a reader expects from the literal they wrote.
+            return round(Decimal(repr(raw)), self._places)
 
     def __repr__(self) -> str:
         places = "" if self._places is None else f", places={self._places}"

django_data_shape-0.2.0/django_data_shape/distributions/zipf.py

@@ -0,0 +1,37 @@
+"""Heavy-tailed weights: a few very large, a long tail of very small."""
+
+from __future__ import annotations
+
+import math
+
+from django_data_shape.invalid_shape import InvalidShape
+
+
+class Zipf:
+    """Positive weights following a power law of exponent ``s``.
+
+    The distribution fan-out is realistically drawn from, and the reason
+    declaring fan-out is worth doing at all. A customer table where every
+    customer has ten orders is not merely tidy, it is the one shape in which the
+    planner is never wrong: its ``n_distinct`` average *is* the truth, so a join
+    estimate cannot miss. Give the head a thousand orders and the tail one, and
+    the same estimate is out by orders of magnitude in both directions -- which
+    is what production looks like and what a test database has to reproduce.
+
+    Inverse transform of a Pareto: ``(1 - draw) ** (-1 / s)``. Larger ``s``
+    means a lighter tail; values near 1 are the classic Zipf regime.
+    """
+
+    def __init__(self, s: float = 1.2) -> None:
+        if not math.isfinite(s) or s <= 0:
+            raise InvalidShape(f"Zipf needs a positive finite exponent, got s={s}.")
+        self._s = s
+
+    def value(self, row: int, draw: float) -> object:
+        # ``1 - draw`` rather than ``draw`` so the singularity sits at the top of
+        # the interval: draw is in [0, 1), so 1 - draw is in (0, 1] and never
+        # zero, which keeps the power finite for every possible draw.
+        return (1.0 - draw) ** (-1.0 / self._s)
+
+    def __repr__(self) -> str:
+        return f"Zipf({self._s!r})"

django_data_shape-0.2.0/django_data_shape/fan_out.py

@@ -0,0 +1,82 @@
+"""How a foreign key's children are spread across their parents."""
+
+from __future__ import annotations
+
+from django_data_shape.distributions.distribution import Distribution
+from django_data_shape.invalid_shape import InvalidShape
+
+PLACEMENTS = ("arrival", "grouped")
+
+
+class FanOut:
+    """A distribution over how many children each parent has.
+
+    Declared as a shape over an **already fixed** child row count, never as a
+    multiplier. Rows times fan-out makes cardinality emergent, and a table whose
+    size is emergent is a table nothing can assert about.
+
+    ``sizes`` supplies a relative weight per parent -- ``Zipf()`` for the
+    realistic heavy tail, ``Uniform(1, 10)`` for something flatter. The weights
+    are normalised, so their scale is irrelevant and only their spread matters.
+
+    ``childless`` is the share of parents with **no** children at all. It is
+    called out separately because it is the case hand-written fixtures always
+    omit and the one that changes what a join does: a parent nobody references
+    is the difference between an inner and an outer join returning the same
+    thing and returning different things.
+
+    ``null`` is the share of children whose foreign key is NULL, which is only
+    meaningful on a nullable column. **It thins the partition uniformly after**
+    it is computed, so ``sizes`` describes the pre-null spread. Stated because
+    the alternative -- partitioning only the non-null children -- would make the
+    declared distribution mean something subtly different from what it says.
+
+    ``placement`` decides where children sit physically, and it is not
+    cosmetic. Emitting them parent by parent gives a perfectly clustered table
+    that no production system has and that flatters every index scan; the
+    default interleaves them the way rows really arrive.
+    """
+
+    def __init__(
+        self,
+        sizes: Distribution,
+        *,
+        childless: float = 0.0,
+        null: float = 0.0,
+        placement: str = "arrival",
+    ) -> None:
+        for name, share in (("childless", childless), ("null", null)):
+            if not 0.0 <= share < 1.0:
+                raise InvalidShape(
+                    f"FanOut {name} is a share of the whole and must be in [0, 1), got {share}."
+                )
+        if placement not in PLACEMENTS:
+            raise InvalidShape(
+                f"FanOut placement must be one of {', '.join(PLACEMENTS)}, got {placement!r}."
+            )
+        self._sizes = sizes
+        self._childless = childless
+        self._null = null
+        self._placement = placement
+
+    @property
+    def sizes(self) -> Distribution:
+        return self._sizes
+
+    @property
+    def childless(self) -> float:
+        return self._childless
+
+    @property
+    def null(self) -> float:
+        return self._null
+
+    @property
+    def placement(self) -> str:
+        return self._placement
+
+    def __repr__(self) -> str:
+        return (
+            f"FanOut({self._sizes!r}, childless={self._childless!r}, "
+            f"null={self._null!r}, placement={self._placement!r})"
+        )

django_data_shape-0.2.0/django_data_shape/fan_out_plan.py

@@ -0,0 +1,75 @@
+"""A fan-out resolved against real parent keys."""
+
+from __future__ import annotations
+
+import bisect
+from math import gcd
+
+from django_data_shape.utils import draw
+
+
+class FanOutPlan:
+    """Which parent owns each child row, decided once and answered in O(1).
+
+    A **partition of the child key range**: parent ``j`` owns rows
+    ``[starts[j], starts[j + 1])``. That representation is the whole point, and
+    it was chosen over the obvious alternative -- drawing a parent per child --
+    because a per-child draw **cannot be inverted**. Asking "which children
+    belong to parent T" is what a mirrored collection needs, and against a draw
+    the only answer is to index every row.
+
+    Two consequences fall out for free. A childless parent is one whose range is
+    empty, so the tail everybody forgets is representable rather than
+    approximated. And physical placement becomes a pure question of the order
+    rows are *emitted* in, entirely separate from which parent owns them -- the
+    same split this design keeps finding between one order and another.
+    """
+
+    def __init__(
+        self,
+        keys: list[int],
+        starts: list[int],
+        rows: int,
+        null_stream: int,
+        null_share: float,
+        interleave: bool,
+    ) -> None:
+        self._keys = keys
+        self._starts = starts
+        self._rows = rows
+        self._null_stream = null_stream
+        self._null_share = null_share
+        self._stride = _stride(rows) if interleave else 1
+
+    def key_for(self, row: int) -> int | None:
+        """The parent key for one child row, or None where the key is null."""
+        if self._null_share and draw(self._null_stream, row) < self._null_share:
+            return None
+        slot = (row * self._stride) % self._rows
+        # bisect_right, not left: a parent with an empty range shares its start
+        # with the next one, and bisect_right steps past every duplicate to the
+        # last parent whose range actually begins at or below the slot. That is
+        # what makes a childless parent unreachable rather than special-cased.
+        return self._keys[bisect.bisect_right(self._starts, slot) - 1]
+
+    def sizes(self) -> list[int]:
+        """How many children each parent ended up with, in parent-key order."""
+        bounds = [*self._starts, self._rows]
+        return [bounds[i + 1] - bounds[i] for i in range(len(self._keys))]
+
+
+def _stride(rows: int) -> int:
+    """A multiplier that walks every slot exactly once, scattering as it goes.
+
+    ``row * stride % rows`` is a bijection precisely when the two are coprime,
+    so children land in an order unrelated to their parent without buffering a
+    permutation or holding any state. Starting near the golden ratio of ``rows``
+    gives the low-discrepancy spread that makes consecutive children come from
+    unrelated parents, which is what "arrival order" means physically.
+    """
+    if rows < 3:
+        return 1
+    candidate = max(2, int(rows * 0.6180339887498949))
+    while gcd(candidate, rows) != 1:
+        candidate += 1
+    return candidate

django_data_shape-0.2.0/django_data_shape/generate_rows.py

@@ -0,0 +1,60 @@
+"""Turning a declared table into the tuples COPY will consume."""
+
+from __future__ import annotations
+
+from collections.abc import Callable, Iterator, Mapping
+from functools import partial
+from typing import Any, cast
+
+from django_data_shape.distributions.distribution import Distribution
+from django_data_shape.fan_out_plan import FanOutPlan
+from django_data_shape.table import Table
+from django_data_shape.utils import draw, field_stream
+
+
+def generate_rows(
+    table: Table, seed: int, plans: Mapping[str, FanOutPlan] | None = None
+) -> Iterator[tuple[Any, ...]]:
+    """Yield one tuple per row: the primary key, then each declared column.
+
+    Rows, not model instances. The ORM is the wrong tool at these counts -- the
+    difference between a load measured in seconds and one measured in minutes --
+    and nothing here needs an instance, because no ``save`` will run and no
+    signal should fire.
+
+    Primary keys are a dense ``1..N`` because this package assigns them, which is
+    what lets a child's foreign key be satisfied without a lookup and what makes
+    a self-referential tree acyclic by construction. It also obliges the caller
+    to reset the sequence afterwards; see ``build``.
+
+    ``plans`` carries the resolved fan-out for each relation column. Resolving
+    happens outside this function because it has to read the parent's real keys
+    out of the database, and keeping the query there leaves generation itself
+    backend-neutral and testable without a connection.
+
+    A generator rather than a list: a million tuples is real memory, psycopg
+    writes them one at a time anyway, and materialising the set would buy
+    nothing but peak RSS.
+    """
+    plans = plans or {}
+    # Each column is reduced to one callable of the row index before the loop
+    # starts. At a million rows the loop body runs a million times per column, so
+    # the branch between a fan-out and a value distribution is worth deciding
+    # once rather than a million times.
+    emit: list[Callable[[int], object]] = []
+    for name, _field in table.columns():
+        plan = plans.get(name)
+        if plan is not None:
+            emit.append(plan.key_for)
+            continue
+        distribution = cast("Distribution", table.fields[name])
+        emit.append(
+            partial(_from_distribution, distribution, field_stream(seed, table.db_table, name))
+        )
+
+    for row in range(table.rows):
+        yield (row + 1, *(produce(row) for produce in emit))
+
+
+def _from_distribution(distribution: Distribution, stream: int, row: int) -> object:
+    return distribution.value(row, draw(stream, row))