django-data-shape 0.12.0__tar.gz → 0.14.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/CHANGELOG.md +138 -1
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/PKG-INFO +1 -1
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/__init__.py +6 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/build.py +59 -9
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/check_constraints.py +35 -10
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/fan_out.py +43 -3
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/fan_out_plan.py +9 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/fan_out_sizes.py +8 -1
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/generate_rows.py +25 -3
- django_data_shape-0.14.0/django_data_shape/keys/disjoint.py +44 -0
- django_data_shape-0.14.0/django_data_shape/keys/md5_keys.py +99 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/keys/uuid_keys.py +11 -0
- django_data_shape-0.14.0/django_data_shape/paired.py +82 -0
- django_data_shape-0.14.0/django_data_shape/paired_plan.py +145 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/projection.py +170 -29
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/resolve_fan_out.py +91 -11
- django_data_shape-0.14.0/django_data_shape/resolve_paired.py +145 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/table.py +46 -3
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/version.py +1 -1
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/keys.md +39 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/projections.md +35 -1
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/reference.md +6 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/relations.md +111 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/pyproject.toml +1 -1
- django_data_shape-0.14.0/tests/keys/test_md5_keys.py +71 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_build.py +46 -1
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_build_derivations.py +28 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_build_graph.py +61 -0
- django_data_shape-0.14.0/tests/test_build_m2m.py +100 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_build_projection.py +100 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_check_constraints.py +82 -0
- django_data_shape-0.14.0/tests/test_fan_out.py +79 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_fan_out_sizes.py +30 -0
- django_data_shape-0.14.0/tests/test_paired.py +34 -0
- django_data_shape-0.14.0/tests/test_paired_plan.py +116 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_projection.py +151 -4
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_resolve_fan_out.py +96 -1
- django_data_shape-0.14.0/tests/test_resolve_paired.py +117 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_table.py +60 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/testapp/models.py +61 -0
- django_data_shape-0.12.0/tests/test_fan_out.py +0 -42
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.github/CODEOWNERS +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.github/SECURITY.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.github/dependabot.yml +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.github/workflows/release.yml +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.github/workflows/tests.yml +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.github/workflows/upstream-drift.yml +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.gitignore +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/.pre-commit-config.yaml +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/CLAUDE.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/LICENSE +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/Makefile +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/README.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/apply_statistics_targets.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/build_result.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/canonical.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/check_invariants.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/children_per_parent.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/clone_database.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivation_queried_database.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/after.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/aligned.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/derivation.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/derived.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/given.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/per_parent.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/derivations/scope.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/ascending.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/bounded.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/categorical.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/constant.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/distinct.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/distribution.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/sequential.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/skew.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/uniform.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/distributions/zipf.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/drop_database.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/fixtures/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/fixtures/scale_fixture.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/fixtures/shape_fixture.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/fixtures/skip_unless_postgres.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/infer_key_strategy.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/invalid_shape.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/invariant.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/invariant_violated.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/keys/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/keys/key_function.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/keys/key_strategy.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/keys/sequential_keys.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/keys/sql_keys.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/order_derivations.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/order_tables.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/py.typed +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/refuse_queries.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/require_clone_strategy.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/require_postgres.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/scale_protocol.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/scaled_shape.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/scaled_world.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/shape.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/shape_digest.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/shape_not_empty.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/table_result.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/template_database.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/unhashable_shape.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/unsupported_backend.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/utils.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/world_changed.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/derivations.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/index.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/invariants.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/pytest.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/docs/statistics.md +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/mkdocs.yml +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/scripts/release-publish.sh +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/conftest.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/conftest_settings.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/derivations/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/derivations/test_after.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/derivations/test_aligned.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/derivations/test_derived.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/derivations/test_given.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/derivations/test_per_parent.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/distributions/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/distributions/test_constant.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/distributions/test_sequential.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/distributions/test_skew.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/distributions/test_uniform.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/distributions/test_zipf.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/keys/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/keys/test_key_function.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/keys/test_sequential_keys.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/keys/test_uuid_keys.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/scale_protocol_consumers.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/scale_protocol_impostors.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_apply_statistics_targets.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_build_invariants.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_build_keys.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_build_portable.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_children_per_parent.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_documentation.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_fixtures.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_generate_rows.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_infer_key_strategy.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_invariant.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_order_derivations.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_order_tables.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_reference_documents_the_public_surface.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_refuse_queries.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_require_clone_strategy.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_require_postgres.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_scale_protocol.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_scaled_shape.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_scaled_world.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_shape.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_shape_digest.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_template_database.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_utils.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/test_version.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/tests/testapp/__init__.py +0 -0
- {django_data_shape-0.12.0 → django_data_shape-0.14.0}/uv.lock +0 -0
|
@@ -7,6 +7,141 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [0.14.0] — 2026-09-03
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- **`Paired` fills the second key of a many-to-many, and the edge count stays
|
|
15
|
+
exact.** Two fan-outs on one through table are refused because a fan-out is a
|
|
16
|
+
partition computed from the row index alone, so two of them partition the same
|
|
17
|
+
rows without either seeing the other -- nothing enumerates the pairs and a
|
|
18
|
+
collision is a matter of the seed. `person=Paired("company", Zipf())` is the
|
|
19
|
+
half that looks: `company` partitions the rows as any fan-out does, and within
|
|
20
|
+
each of its groups `person` takes that many **distinct** partners. A duplicate
|
|
21
|
+
pair is impossible by construction rather than removed afterwards, so nothing
|
|
22
|
+
is deduplicated and the build never reports an achieved count against a
|
|
23
|
+
requested one. The design notes for this milestone had given that up; they
|
|
24
|
+
were wrong, and the rule that cardinality is declared survives M2M.
|
|
25
|
+
- **What binds is the busiest group, not the product.** Every row of one
|
|
26
|
+
group needs a different partner, so the constraint is
|
|
27
|
+
`largest group <= partners` rather than `rows <= groups x partners` -- and a
|
|
28
|
+
heavy tail puts a large share of every edge on one group, `Zipf(1.2)` over
|
|
29
|
+
five thousand of them putting 21% on the top one. A declaration that looks
|
|
30
|
+
sparse against the product can still be impossible, and it cannot be known
|
|
31
|
+
until the partition is resolved: this is the one structural refusal that
|
|
32
|
+
waits for the build rather than firing where the shape is written.
|
|
33
|
+
- **The second side is derived and approximate, and the numbers are
|
|
34
|
+
published.** Both marginals plus the edge count is over-determined, and
|
|
35
|
+
fixing all three is a constraint satisfaction problem that cannot stream
|
|
36
|
+
into `COPY`. Measured against exact weighted sampling over 200,000 edges,
|
|
37
|
+
the busiest partner comes out 1.09 times as busy, the 99th percentile about
|
|
38
|
+
30% high, and about 7% fewer partners are touched.
|
|
39
|
+
- **Partners are allocated across weight bands and strided within one**, so
|
|
40
|
+
nothing is ever drawn and asked whether it is taken. Rejection sampling
|
|
41
|
+
degenerates exactly where the shape is most interesting -- 245 probes per
|
|
42
|
+
row on that world -- and the usual fix, sampling the ones to leave out when
|
|
43
|
+
a group wants more than half, is ten times faster and **a different sampling
|
|
44
|
+
rule**, so a group one row over the halfway mark would get a materially
|
|
45
|
+
different membership from one row under. The band count is **derived from
|
|
46
|
+
the declared weights** rather than chosen, which systematic allocation is
|
|
47
|
+
what makes possible: it converges as bands get finer where largest-remainder
|
|
48
|
+
runs away, so "enough" is a limit rather than a number somebody tuned.
|
|
49
|
+
|
|
50
|
+
- **`FanOutPlan.group_of`** answers which parent a row belongs to, which a
|
|
51
|
+
pairing needs to choose partners inside this row's group.
|
|
52
|
+
|
|
53
|
+
- **`Projection(..., through=Model)` says which model a derived join runs on.**
|
|
54
|
+
An abstract base carrying `created_by`/`updated_by` to `User` -- a pattern most
|
|
55
|
+
Django schemas have -- makes **any two models in the schema joinable**, so the
|
|
56
|
+
derivation had candidates everywhere and could settle nothing: not for one
|
|
57
|
+
pair but for every pair, which put the derived form out of reach for a whole
|
|
58
|
+
application and left `sql=` as the only route to this feature's own motivating
|
|
59
|
+
example. The refusal now names `through=` first, and names a model that can
|
|
60
|
+
actually resolve the join rather than the first one alphabetically -- an audit
|
|
61
|
+
model is reached by two edges from each side, so it is a real candidate and
|
|
62
|
+
never a usable answer. Where every candidate is reached more than once,
|
|
63
|
+
`through=` cannot narrow anything and the refusal says so instead of
|
|
64
|
+
suggesting it.
|
|
65
|
+
|
|
66
|
+
### Changed
|
|
67
|
+
|
|
68
|
+
- **The two-fan-out refusal names the declaration that replaces it.** Its remedy
|
|
69
|
+
used to be a statement of your own, because nothing else could keep the
|
|
70
|
+
constraint. It now names `Paired` first and keeps `Projection(columns=, sql=)`
|
|
71
|
+
as the escape hatch for an edge set that has to be a particular one.
|
|
72
|
+
|
|
73
|
+
### Fixed
|
|
74
|
+
|
|
75
|
+
- **A rounded `Uniform` can size a fan-out.**
|
|
76
|
+
`FanOut(Uniform(1, 10, places=0))` was refused with "needs numeric fan-out
|
|
77
|
+
sizes, but produced `Decimal('3')`" -- which reads as a contradiction, and was
|
|
78
|
+
one keyword away from `FanOut(Uniform(1, 10))`, the spelling this package's own
|
|
79
|
+
docstring recommends and two of its refusals suggest. `places=0` is the natural
|
|
80
|
+
way to say that a fan-out size is a count. The guard was
|
|
81
|
+
`isinstance(weight, (int, float))` while the next line already did
|
|
82
|
+
`float(weight)`, so it was stricter than the arithmetic it protected. It is now
|
|
83
|
+
the numeric tower plus `Decimal`, which `numbers.Real` deliberately excludes,
|
|
84
|
+
and `Fraction` and numpy scalars come along with it. `bool` stays refused. The
|
|
85
|
+
message also names a remedy now, which it was alone in not doing.
|
|
86
|
+
|
|
87
|
+
- **`columns=` accepts the column an `INSERT` actually lists.** The refusal that
|
|
88
|
+
sends a reader there says "the select's columns are what the insert lists", and
|
|
89
|
+
what an insert lists for a foreign key is `event_id` -- which was then refused
|
|
90
|
+
as "no field named event_id". That is a contradiction rather than a correction,
|
|
91
|
+
and the surrounding documentation reinforced the wrong reading. Both spellings
|
|
92
|
+
are accepted and resolve to the same column, the messages say so, and the same
|
|
93
|
+
dict is read for `statistics=` so the two cannot disagree about what a column
|
|
94
|
+
is called.
|
|
95
|
+
|
|
96
|
+
## [0.13.0] — 2026-09-03
|
|
97
|
+
|
|
98
|
+
### Added
|
|
99
|
+
- **`FanOut(parents=[...])` narrows a fan-out to the parents it names**, which
|
|
100
|
+
is the only gap two independent consumers reached separately: a tenant-scoped
|
|
101
|
+
schema could not pin a foreign key to one pre-existing company, and a
|
|
102
|
+
`GenericForeignKey` fanned out over every row of `django_content_type`. Real
|
|
103
|
+
keys, narrowed by the database rather than filtered afterwards, and a named
|
|
104
|
+
key that matches no row is refused **by key** rather than quietly taking a
|
|
105
|
+
smaller share -- that failure is silent otherwise, because the rows that would
|
|
106
|
+
have gone to it go to the others instead, or nowhere if it was the only one.
|
|
107
|
+
A list of keys and never a queryset: a declaration needs no connection, and
|
|
108
|
+
evaluating one here would give it one.
|
|
109
|
+
- It reaches the arithmetic as well as the load. Capacity for a narrowed
|
|
110
|
+
column is the **named** count, so the pigeonhole no longer overstates the
|
|
111
|
+
room by the ratio of the two, and the same substitution makes the
|
|
112
|
+
no-collision exemption read the named count. `fan_out_sizes` no longer
|
|
113
|
+
reports `WorldChanged` for a narrowed partition, because a parent table
|
|
114
|
+
holding more rows than the partition covers is the declaration working --
|
|
115
|
+
and a named key that has since gone is refused earlier, by name.
|
|
116
|
+
- **The documentation argues against the obvious use**, because the obvious
|
|
117
|
+
use is against this package's own thesis: a column pinned to a single parent
|
|
118
|
+
has `n_distinct = 1`, so a tenant filter is priced at nothing and looks free
|
|
119
|
+
in a test database and does not in production. The recipe is several tenants
|
|
120
|
+
with yours among the heavy ones, reached through `fan_out_sizes`.
|
|
121
|
+
- **`Md5Keys` is a UUID key strategy with a SQL twin**, so a projection can fill
|
|
122
|
+
a UUID-keyed table. A projection has no declared row count, so its rows never
|
|
123
|
+
pass through Python and its keys are written by the statement that inserts
|
|
124
|
+
them -- and `blake2b` has no PostgreSQL equivalent, which is where every
|
|
125
|
+
UUID-keyed projection used to stop. `md5` exists on both sides. **A separate
|
|
126
|
+
strategy, never a second meaning for `UuidKeys`**: the two draw different keys
|
|
127
|
+
for the same row, so substituting one for the other would change every key in
|
|
128
|
+
every world already built. The two halves are compared against a real server
|
|
129
|
+
rather than against a checked-in fixture, which would only prove the generator
|
|
130
|
+
had not changed. The projection refusal now names it as the remedy.
|
|
131
|
+
- **`Disjoint`, the seventh opt-in protocol**, which is how a key strategy says
|
|
132
|
+
its keys cannot collide with rows already in the table.
|
|
133
|
+
|
|
134
|
+
### Fixed
|
|
135
|
+
- **A UUID-keyed table can be built beside rows your own code made.** Building
|
|
136
|
+
over a non-empty table is refused because this package assigns keys from 1 and
|
|
137
|
+
a second build collides -- and that sentence is about integer keys, where the
|
|
138
|
+
refusal was not. Both UUID strategies derive a 128-bit digest per row and
|
|
139
|
+
cannot land on a factory's row, so the refusal blocked the hybrid this package
|
|
140
|
+
documents in exactly the schemas where UUID primary keys are the norm.
|
|
141
|
+
`SequentialKeys` still collides and is still refused, and `KeyFunction` is
|
|
142
|
+
read as able to collide because this package cannot read your function.
|
|
143
|
+
|
|
144
|
+
|
|
10
145
|
## [0.12.0] — 2026-09-03
|
|
11
146
|
|
|
12
147
|
### Fixed
|
|
@@ -832,7 +967,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
832
967
|
into `COPY FROM STDIN`, which psycopg 2 cannot do without materialising them
|
|
833
968
|
first.
|
|
834
969
|
|
|
835
|
-
[Unreleased]: https://github.com/Artui/django-data-shape/compare/v0.
|
|
970
|
+
[Unreleased]: https://github.com/Artui/django-data-shape/compare/v0.14.0...HEAD
|
|
971
|
+
[0.14.0]: https://github.com/Artui/django-data-shape/compare/v0.13.0...v0.14.0
|
|
972
|
+
[0.13.0]: https://github.com/Artui/django-data-shape/compare/v0.12.0...v0.13.0
|
|
836
973
|
[0.12.0]: https://github.com/Artui/django-data-shape/compare/v0.11.0...v0.12.0
|
|
837
974
|
[0.11.0]: https://github.com/Artui/django-data-shape/compare/v0.10.0...v0.11.0
|
|
838
975
|
[0.10.0]: https://github.com/Artui/django-data-shape/compare/v0.9.0...v0.10.0
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: django-data-shape
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.14.0
|
|
4
4
|
Summary: A realistically shaped test database from Django models: declare cardinality, skew and fan-out, load by COPY, and make the query planner believe it.
|
|
5
5
|
Project-URL: Homepage, https://github.com/Artui/django-data-shape
|
|
6
6
|
Project-URL: Repository, https://github.com/Artui/django-data-shape
|
|
@@ -32,11 +32,14 @@ from django_data_shape.fan_out_sizes import fan_out_sizes
|
|
|
32
32
|
from django_data_shape.invalid_shape import InvalidShape
|
|
33
33
|
from django_data_shape.invariant import Invariant
|
|
34
34
|
from django_data_shape.invariant_violated import InvariantViolated
|
|
35
|
+
from django_data_shape.keys.disjoint import Disjoint
|
|
35
36
|
from django_data_shape.keys.key_function import KeyFunction
|
|
36
37
|
from django_data_shape.keys.key_strategy import KeyStrategy
|
|
38
|
+
from django_data_shape.keys.md5_keys import Md5Keys
|
|
37
39
|
from django_data_shape.keys.sequential_keys import SequentialKeys
|
|
38
40
|
from django_data_shape.keys.sql_keys import SqlKeys
|
|
39
41
|
from django_data_shape.keys.uuid_keys import UuidKeys
|
|
42
|
+
from django_data_shape.paired import Paired
|
|
40
43
|
from django_data_shape.projection import Projection
|
|
41
44
|
from django_data_shape.require_clone_strategy import require_clone_strategy
|
|
42
45
|
from django_data_shape.scale_protocol import ScaleProtocol
|
|
@@ -66,6 +69,7 @@ __all__ = [
|
|
|
66
69
|
"Derivation",
|
|
67
70
|
"DerivationQueriedDatabase",
|
|
68
71
|
"Derived",
|
|
72
|
+
"Disjoint",
|
|
69
73
|
"Distinct",
|
|
70
74
|
"Distribution",
|
|
71
75
|
"FanOut",
|
|
@@ -74,7 +78,9 @@ __all__ = [
|
|
|
74
78
|
"Invariant",
|
|
75
79
|
"InvariantViolated",
|
|
76
80
|
"KeyFunction",
|
|
81
|
+
"Md5Keys",
|
|
77
82
|
"KeyStrategy",
|
|
83
|
+
"Paired",
|
|
78
84
|
"PerParent",
|
|
79
85
|
"Projection",
|
|
80
86
|
"ScaleProtocol",
|
|
@@ -19,11 +19,15 @@ from django_data_shape.fan_out import FanOut
|
|
|
19
19
|
from django_data_shape.fan_out_plan import FanOutPlan
|
|
20
20
|
from django_data_shape.generate_rows import generate_rows
|
|
21
21
|
from django_data_shape.invalid_shape import InvalidShape
|
|
22
|
+
from django_data_shape.keys.disjoint import Disjoint
|
|
22
23
|
from django_data_shape.order_tables import order_tables
|
|
24
|
+
from django_data_shape.paired import Paired
|
|
25
|
+
from django_data_shape.paired_plan import PairedPlan
|
|
23
26
|
from django_data_shape.projection import Projection
|
|
24
27
|
from django_data_shape.refuse_queries import refuse_queries
|
|
25
28
|
from django_data_shape.require_postgres import require_postgres
|
|
26
29
|
from django_data_shape.resolve_fan_out import resolve_fan_out
|
|
30
|
+
from django_data_shape.resolve_paired import resolve_paired
|
|
27
31
|
from django_data_shape.shape import Shape
|
|
28
32
|
from django_data_shape.shape_not_empty import ShapeNotEmpty
|
|
29
33
|
from django_data_shape.table import Table
|
|
@@ -98,7 +102,7 @@ def build(
|
|
|
98
102
|
# its parent's real keys and a projection selects from whole tables, and
|
|
99
103
|
# a table with no rows yet has neither.
|
|
100
104
|
for table in order_tables(shape.tables):
|
|
101
|
-
_require_empty(connection, table
|
|
105
|
+
_require_empty(connection, table)
|
|
102
106
|
# Before the rows rather than beside the ANALYZE, for two reasons
|
|
103
107
|
# that point the same way. A target changed after ANALYZE has run
|
|
104
108
|
# does nothing until the next one, so it has to be in place before
|
|
@@ -114,8 +118,8 @@ def build(
|
|
|
114
118
|
if isinstance(table, Projection):
|
|
115
119
|
loaded = _project(connection, table, shape.seed)
|
|
116
120
|
else:
|
|
117
|
-
plans = _resolve(connection, table, shape.seed)
|
|
118
|
-
loaded = _load(connection, table, shape.seed, plans)
|
|
121
|
+
plans, pairings = _resolve(connection, table, shape.seed)
|
|
122
|
+
loaded = _load(connection, table, shape.seed, plans, pairings)
|
|
119
123
|
_reset_sequence(connection, table.model)
|
|
120
124
|
_analyze(connection, table.db_table)
|
|
121
125
|
results.append(TableResult(table=table.db_table, rows=loaded))
|
|
@@ -162,11 +166,21 @@ def _project(connection: Any, projection: Projection, seed: int) -> int:
|
|
|
162
166
|
return inserted
|
|
163
167
|
|
|
164
168
|
|
|
165
|
-
def _resolve(
|
|
166
|
-
|
|
169
|
+
def _resolve(
|
|
170
|
+
connection: Any, table: Table, seed: int
|
|
171
|
+
) -> tuple[dict[str, FanOutPlan], dict[str, PairedPlan]]:
|
|
172
|
+
"""Partition each declared relation over the parent keys that exist.
|
|
173
|
+
|
|
174
|
+
Fan-outs first and pairings second, because a pairing chooses partners
|
|
175
|
+
*inside* the groups a fan-out produced -- so the partition it reads has to
|
|
176
|
+
exist before it can be resolved. That ordering is the mechanism rather than
|
|
177
|
+
a detail of this function: it is what makes an edge table one pass.
|
|
178
|
+
"""
|
|
167
179
|
parent_fields = table.parent_fields()
|
|
168
180
|
plans: dict[str, FanOutPlan] = {}
|
|
169
181
|
for name, field in table.relations():
|
|
182
|
+
if isinstance(table.fields[name], Paired):
|
|
183
|
+
continue
|
|
170
184
|
plans[name] = resolve_fan_out(
|
|
171
185
|
cast("FanOut", table.fields[name]),
|
|
172
186
|
cast("type[Model]", field.related_model),
|
|
@@ -177,10 +191,25 @@ def _resolve(connection: Any, table: Table, seed: int) -> dict[str, FanOutPlan]:
|
|
|
177
191
|
connection,
|
|
178
192
|
parent_fields.get(name, ()),
|
|
179
193
|
)
|
|
180
|
-
|
|
194
|
+
pairings: dict[str, PairedPlan] = {}
|
|
195
|
+
for name, field in table.relations():
|
|
196
|
+
declared = table.fields[name]
|
|
197
|
+
if not isinstance(declared, Paired):
|
|
198
|
+
continue
|
|
199
|
+
pairings[name] = resolve_paired(
|
|
200
|
+
declared,
|
|
201
|
+
cast("type[Model]", field.related_model),
|
|
202
|
+
plans[declared.relation],
|
|
203
|
+
table.rows,
|
|
204
|
+
seed,
|
|
205
|
+
table.db_table,
|
|
206
|
+
name,
|
|
207
|
+
connection,
|
|
208
|
+
)
|
|
209
|
+
return plans, pairings
|
|
181
210
|
|
|
182
211
|
|
|
183
|
-
def _require_empty(connection: Any,
|
|
212
|
+
def _require_empty(connection: Any, table: Table | Projection) -> None:
|
|
184
213
|
"""Refuse to build on top of rows that are already there.
|
|
185
214
|
|
|
186
215
|
The keys this package assigns start at 1 every time, so a second build over
|
|
@@ -188,6 +217,17 @@ def _require_empty(connection: Any, db_table: str) -> None:
|
|
|
188
217
|
UniqueViolation naming an index, which tells the reader nothing about what
|
|
189
218
|
they did or what to do instead.
|
|
190
219
|
|
|
220
|
+
**Unless the key strategy says it cannot collide**, which is
|
|
221
|
+
:class:`~django_data_shape.keys.disjoint.Disjoint` and is why the refusal is
|
|
222
|
+
asked rather than assumed. The sentence above is about integer keys and the
|
|
223
|
+
refusal was not, so a ``UuidKeys`` table -- a 128-bit digest per row, unable
|
|
224
|
+
to land on a caller's row -- was refused for a collision it cannot have.
|
|
225
|
+
That blocked the hybrid this package documents, parents made by your code
|
|
226
|
+
and children made here, in exactly the schemas where UUID keys are the norm.
|
|
227
|
+
|
|
228
|
+
A ``Projection`` has no key strategy of its own to ask, and its rows come
|
|
229
|
+
from a statement rather than from generated keys, so it takes the refusal.
|
|
230
|
+
|
|
191
231
|
The message names the likely cause and not only the remedy, because the
|
|
192
232
|
first consumer met this from a direction the remedy does not fit: a
|
|
193
233
|
session-scoped ``shape_fixture`` and a scaled world pointed at one model.
|
|
@@ -195,6 +235,10 @@ def _require_empty(connection: Any, db_table: str) -> None:
|
|
|
195
235
|
never mentions -- so "empty the table first" reads as advice about somebody
|
|
196
236
|
else's data.
|
|
197
237
|
"""
|
|
238
|
+
keys = getattr(table, "keys", None)
|
|
239
|
+
if isinstance(keys, Disjoint) and keys.is_disjoint_from_existing_rows():
|
|
240
|
+
return
|
|
241
|
+
db_table = table.db_table
|
|
198
242
|
with connection.cursor() as cursor:
|
|
199
243
|
cursor.execute(f"SELECT EXISTS (SELECT 1 FROM {connection.ops.quote_name(db_table)})")
|
|
200
244
|
row = cursor.fetchone()
|
|
@@ -212,7 +256,13 @@ def _require_empty(connection: Any, db_table: str) -> None:
|
|
|
212
256
|
# API reached through Django's cursor wrapper, and it appears on no Django base
|
|
213
257
|
# class, so annotating the real wrapper type would mean asserting the checker
|
|
214
258
|
# out of the way on every line that uses it.
|
|
215
|
-
def _load(
|
|
259
|
+
def _load(
|
|
260
|
+
connection: Any,
|
|
261
|
+
table: Table,
|
|
262
|
+
seed: int,
|
|
263
|
+
plans: dict[str, FanOutPlan],
|
|
264
|
+
pairings: dict[str, PairedPlan],
|
|
265
|
+
) -> int:
|
|
216
266
|
"""Stream generated rows into the table, by the fastest route the backend has.
|
|
217
267
|
|
|
218
268
|
Each declared value is passed through its field's ``get_db_prep_save``
|
|
@@ -238,7 +288,7 @@ def _load(connection: Any, table: Table, seed: int, plans: dict[str, FanOutPlan]
|
|
|
238
288
|
prepare = [pk_field.get_db_prep_save] + [field.get_db_prep_save for _, field in table.columns()]
|
|
239
289
|
rows = (
|
|
240
290
|
tuple(prep(value, connection) for prep, value in zip(prepare, row, strict=True))
|
|
241
|
-
for row in generate_rows(table, seed, plans)
|
|
291
|
+
for row in generate_rows(table, seed, plans, pairings)
|
|
242
292
|
)
|
|
243
293
|
|
|
244
294
|
# The guard is built here rather than inside each route because it belongs
|
{django_data_shape-0.12.0 → django_data_shape-0.14.0}/django_data_shape/check_constraints.py
RENAMED
|
@@ -163,8 +163,12 @@ def _check_independent_fan_outs(
|
|
|
163
163
|
That is why this is otherwise a refusal and not a wider capacity
|
|
164
164
|
calculation: no arithmetic over the two marginals decides it, because the
|
|
165
165
|
failure is that nothing enumerates the combinations at all. Deduplicated edges are a
|
|
166
|
-
different algorithm rather than a bigger loop
|
|
167
|
-
|
|
166
|
+
different algorithm rather than a bigger loop. **That algorithm now exists**
|
|
167
|
+
as :class:`~django_data_shape.paired.Paired`, which is the remedy this
|
|
168
|
+
refusal names: one side stays a fan-out and the other takes distinct
|
|
169
|
+
partners inside its groups, so a duplicate pair is impossible rather than
|
|
170
|
+
removed. A statement of your own is still the answer for an edge set that
|
|
171
|
+
has to be a particular one --
|
|
168
172
|
:class:`~django_data_shape.projection.Projection` with ``columns=`` and
|
|
169
173
|
``sql=`` fills the table from a select that can deduplicate.
|
|
170
174
|
"""
|
|
@@ -191,10 +195,11 @@ def _check_independent_fan_outs(
|
|
|
191
195
|
"pairs come out together is an artefact of that index, and two rows sharing a pair is a "
|
|
192
196
|
"matter of the seed. The combinations do fit, which is why the pigeonhole arithmetic "
|
|
193
197
|
"lets this through -- what is missing is anything that enumerates them, and the load "
|
|
194
|
-
"then fails inside COPY at a row number that moves when the seed does.
|
|
195
|
-
f"
|
|
196
|
-
"
|
|
197
|
-
"
|
|
198
|
+
"then fails inside COPY at a row number that moves when the seed does. An edge table "
|
|
199
|
+
f"declares one side and pairs the other: {fanned[1]}=Paired({fanned[0]!r}, Zipf()) takes "
|
|
200
|
+
f"distinct partners inside each {fanned[0]} group, so no pair can repeat and the edge "
|
|
201
|
+
"count is still exactly the row count. An edge set that has to be a particular one is "
|
|
202
|
+
f"filled by a statement instead -- Projection({where}, columns=(...), sql=...)."
|
|
198
203
|
)
|
|
199
204
|
|
|
200
205
|
|
|
@@ -454,8 +459,7 @@ def _cannot_collide(table: Table, column: str, rows_of: dict[type[Model], int])
|
|
|
454
459
|
and the two-fan-out pair beside it.
|
|
455
460
|
"""
|
|
456
461
|
fan_out = cast("FanOut", table.fields[column])
|
|
457
|
-
|
|
458
|
-
parent_rows = rows_of.get(parent)
|
|
462
|
+
parent_rows = _parents_available(table, column, fan_out, rows_of)
|
|
459
463
|
return (
|
|
460
464
|
isinstance(fan_out.sizes, Bounded)
|
|
461
465
|
and fan_out.sizes.distinct_values() == 1
|
|
@@ -561,8 +565,7 @@ def _capacity(table: Table, fields: tuple[str, ...], rows_of: dict[type[Model],
|
|
|
561
565
|
# those rows are exempt from the constraint entirely.
|
|
562
566
|
if declared.null:
|
|
563
567
|
return None
|
|
564
|
-
|
|
565
|
-
parent_rows = rows_of.get(parent)
|
|
568
|
+
parent_rows = _parents_available(table, name, declared, rows_of)
|
|
566
569
|
if parent_rows is None:
|
|
567
570
|
return None
|
|
568
571
|
total *= parent_rows
|
|
@@ -573,6 +576,28 @@ def _capacity(table: Table, fields: tuple[str, ...], rows_of: dict[type[Model],
|
|
|
573
576
|
return total
|
|
574
577
|
|
|
575
578
|
|
|
579
|
+
def _parents_available(
|
|
580
|
+
table: Table, column: str, fan_out: FanOut, rows_of: dict[type[Model], int]
|
|
581
|
+
) -> int | None:
|
|
582
|
+
"""How many parents this fan-out can actually point at, or None if unknown.
|
|
583
|
+
|
|
584
|
+
``parents=`` is the answer where it is given, and it is a **better** answer
|
|
585
|
+
than the parent's row count rather than a substitute for it: it is known
|
|
586
|
+
without the parent being declared in this shape at all, and it is what the
|
|
587
|
+
partition will really spread over. Reading the table's rows instead would
|
|
588
|
+
overstate the room by the ratio of the two -- fifty companies declared and
|
|
589
|
+
one pinned -- and an overstated capacity is a pigeonhole that fails to refuse
|
|
590
|
+
a shape which cannot fit.
|
|
591
|
+
|
|
592
|
+
Without it, the parent has to be declared here: a parent loaded by the caller
|
|
593
|
+
has however many rows it has, and this runs before anything is built.
|
|
594
|
+
"""
|
|
595
|
+
if fan_out.parents is not None:
|
|
596
|
+
return len(fan_out.parents)
|
|
597
|
+
parent = cast("type[Model]", table.model._meta.get_field(column).related_model)
|
|
598
|
+
return rows_of.get(parent)
|
|
599
|
+
|
|
600
|
+
|
|
576
601
|
def _produces_any(declared: object, values: tuple[object, ...]) -> bool | None:
|
|
577
602
|
"""Whether this declaration can put **any** of these in a row.
|
|
578
603
|
|
|
@@ -2,6 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
from __future__ import annotations
|
|
4
4
|
|
|
5
|
+
from collections.abc import Sequence
|
|
6
|
+
|
|
5
7
|
from django_data_shape.distributions.distribution import Distribution
|
|
6
8
|
from django_data_shape.invalid_shape import InvalidShape
|
|
7
9
|
|
|
@@ -52,6 +54,7 @@ class FanOut:
|
|
|
52
54
|
self,
|
|
53
55
|
sizes: Distribution,
|
|
54
56
|
*,
|
|
57
|
+
parents: Sequence[object] | None = None,
|
|
55
58
|
childless: float = 0.0,
|
|
56
59
|
null: float = 0.0,
|
|
57
60
|
placement: str = "arrival",
|
|
@@ -65,7 +68,32 @@ class FanOut:
|
|
|
65
68
|
raise InvalidShape(
|
|
66
69
|
f"FanOut placement must be one of {', '.join(PLACEMENTS)}, got {placement!r}."
|
|
67
70
|
)
|
|
71
|
+
narrowed = None if parents is None else tuple(parents)
|
|
72
|
+
if narrowed is not None:
|
|
73
|
+
# An empty sequence is falsy, and reading it as "not declared" would
|
|
74
|
+
# spread the table over every parent there is -- the opposite of what
|
|
75
|
+
# was asked, silently, in the one argument whose whole purpose is
|
|
76
|
+
# narrowing. So the two are told apart by `is None` and this is a
|
|
77
|
+
# refusal rather than a default.
|
|
78
|
+
if not narrowed:
|
|
79
|
+
raise InvalidShape(
|
|
80
|
+
"FanOut parents= names no parent, so there is nothing to spread these rows "
|
|
81
|
+
"over. Leave it out to use every parent in the table, or name at least one."
|
|
82
|
+
)
|
|
83
|
+
# Not deduplicated: a key named twice weighs that parent twice, so
|
|
84
|
+
# the partition would not be the one written down, and the likeliest
|
|
85
|
+
# cause is a list built by a loop that ran once too often.
|
|
86
|
+
seen: set[object] = set()
|
|
87
|
+
for key in narrowed:
|
|
88
|
+
if key in seen:
|
|
89
|
+
raise InvalidShape(
|
|
90
|
+
f"FanOut parents= names {key!r} more than once. A repeated key weighs "
|
|
91
|
+
"that parent twice, which is a different partition from the one this "
|
|
92
|
+
"declaration reads as."
|
|
93
|
+
)
|
|
94
|
+
seen.add(key)
|
|
68
95
|
self._sizes = sizes
|
|
96
|
+
self._parents = narrowed
|
|
69
97
|
self._childless = childless
|
|
70
98
|
self._null = null
|
|
71
99
|
self._placement = placement
|
|
@@ -74,6 +102,10 @@ class FanOut:
|
|
|
74
102
|
def sizes(self) -> Distribution:
|
|
75
103
|
return self._sizes
|
|
76
104
|
|
|
105
|
+
@property
|
|
106
|
+
def parents(self) -> tuple[object, ...] | None:
|
|
107
|
+
return self._parents
|
|
108
|
+
|
|
77
109
|
@property
|
|
78
110
|
def childless(self) -> float:
|
|
79
111
|
return self._childless
|
|
@@ -87,11 +119,19 @@ class FanOut:
|
|
|
87
119
|
return self._placement
|
|
88
120
|
|
|
89
121
|
def canonical(self) -> object:
|
|
90
|
-
"""The size distribution
|
|
91
|
-
|
|
122
|
+
"""The size distribution, the parents it spreads over, and the shares. See ``Canonical``.
|
|
123
|
+
|
|
124
|
+
``parents`` belongs here rather than being read as a filter applied
|
|
125
|
+
afterwards: two shapes differing only in which parents they cover are
|
|
126
|
+
two different worlds, and the template-database cache keys on this. A
|
|
127
|
+
clone built for one tenant handed to a test asking for another would be
|
|
128
|
+
the cache serving a world nobody declared.
|
|
129
|
+
"""
|
|
130
|
+
return (self._sizes, self._parents, self._childless, self._null, self._placement)
|
|
92
131
|
|
|
93
132
|
def __repr__(self) -> str:
|
|
133
|
+
narrowed = "" if self._parents is None else f"parents={self._parents!r}, "
|
|
94
134
|
return (
|
|
95
|
-
f"FanOut({self._sizes!r}, childless={self._childless!r}, "
|
|
135
|
+
f"FanOut({self._sizes!r}, {narrowed}childless={self._childless!r}, "
|
|
96
136
|
f"null={self._null!r}, placement={self._placement!r})"
|
|
97
137
|
)
|
|
@@ -101,6 +101,15 @@ class FanOutPlan:
|
|
|
101
101
|
end = self._starts[index + 1] if index + 1 < len(self._starts) else self._rows
|
|
102
102
|
return slot - start, end - start
|
|
103
103
|
|
|
104
|
+
def group_of(self, row: int) -> int:
|
|
105
|
+
"""Which parent this row belongs to, as an index into the parent keys.
|
|
106
|
+
|
|
107
|
+
Public because a paired column needs it: the partner is chosen inside
|
|
108
|
+
this row's group, so "which group" and "where in it" are both asked of
|
|
109
|
+
the partition rather than recomputed beside it.
|
|
110
|
+
"""
|
|
111
|
+
return self._slot(row)
|
|
112
|
+
|
|
104
113
|
def _slot(self, row: int) -> int:
|
|
105
114
|
"""Which parent owns this child row, as an index into ``keys``."""
|
|
106
115
|
slot = (row * self._stride) % self._rows
|
|
@@ -114,7 +114,14 @@ def fan_out_sizes(
|
|
|
114
114
|
)
|
|
115
115
|
keys = plan.parent_keys()
|
|
116
116
|
|
|
117
|
-
|
|
117
|
+
# A narrowed fan-out covers the keys it named and no others, so the parent
|
|
118
|
+
# table holding more rows than the partition covers is the declaration
|
|
119
|
+
# working rather than the world moving. The failure WorldChanged exists for
|
|
120
|
+
# is still caught, and caught earlier: a named key that is no longer a row is
|
|
121
|
+
# refused by name when the partition is resolved, above.
|
|
122
|
+
parent_declaration = (
|
|
123
|
+
None if fan_out.parents is not None else declared.get(parent._meta.db_table)
|
|
124
|
+
)
|
|
118
125
|
if isinstance(parent_declaration, Table) and len(keys) != parent_declaration.rows:
|
|
119
126
|
raise WorldChanged(
|
|
120
127
|
f"{parent.__name__} holds {len(keys)} rows and this shape declares "
|
|
@@ -10,6 +10,8 @@ from django_data_shape.derivations.derivation import Derivation
|
|
|
10
10
|
from django_data_shape.derivations.scope import Scope
|
|
11
11
|
from django_data_shape.distributions.distribution import Distribution
|
|
12
12
|
from django_data_shape.fan_out_plan import FanOutPlan
|
|
13
|
+
from django_data_shape.paired import Paired
|
|
14
|
+
from django_data_shape.paired_plan import PairedPlan
|
|
13
15
|
from django_data_shape.table import Table
|
|
14
16
|
from django_data_shape.utils import draw, field_stream
|
|
15
17
|
|
|
@@ -20,7 +22,10 @@ _Source = Callable[[int, list[object]], object]
|
|
|
20
22
|
|
|
21
23
|
|
|
22
24
|
def generate_rows(
|
|
23
|
-
table: Table,
|
|
25
|
+
table: Table,
|
|
26
|
+
seed: int,
|
|
27
|
+
plans: Mapping[str, FanOutPlan] | None = None,
|
|
28
|
+
pairings: Mapping[str, PairedPlan] | None = None,
|
|
24
29
|
) -> Iterator[tuple[Any, ...]]:
|
|
25
30
|
"""Yield one tuple per row: the primary key, then each declared column.
|
|
26
31
|
|
|
@@ -54,6 +59,7 @@ def generate_rows(
|
|
|
54
59
|
nothing but peak RSS.
|
|
55
60
|
"""
|
|
56
61
|
plans = plans or {}
|
|
62
|
+
pairings = pairings or {}
|
|
57
63
|
keys = table.keys
|
|
58
64
|
key_stream = field_stream(seed, table.db_table, ":key")
|
|
59
65
|
columns = table.columns()
|
|
@@ -64,7 +70,7 @@ def generate_rows(
|
|
|
64
70
|
# of column this is, and where each of its sources comes from, are decided
|
|
65
71
|
# once rather than a million times.
|
|
66
72
|
steps: list[_Step] = [
|
|
67
|
-
(slot_of[name], _producer(table, seed, plans, name))
|
|
73
|
+
(slot_of[name], _producer(table, seed, plans, pairings, name))
|
|
68
74
|
for name, _field in columns
|
|
69
75
|
if name not in table.computation_order()
|
|
70
76
|
]
|
|
@@ -81,12 +87,23 @@ def generate_rows(
|
|
|
81
87
|
|
|
82
88
|
|
|
83
89
|
def _producer(
|
|
84
|
-
table: Table,
|
|
90
|
+
table: Table,
|
|
91
|
+
seed: int,
|
|
92
|
+
plans: Mapping[str, FanOutPlan],
|
|
93
|
+
pairings: Mapping[str, PairedPlan],
|
|
94
|
+
name: str,
|
|
85
95
|
) -> Callable[[int, list[object]], object]:
|
|
86
96
|
"""The step for a column that depends on nothing already in the row."""
|
|
87
97
|
plan = plans.get(name)
|
|
88
98
|
if plan is not None:
|
|
89
99
|
return partial(_from_plan, plan)
|
|
100
|
+
pairing = pairings.get(name)
|
|
101
|
+
if pairing is not None:
|
|
102
|
+
# Two questions of the fan-out this pairs with -- which group, and where
|
|
103
|
+
# in it -- and one of the pairing. All three are O(1) because both are
|
|
104
|
+
# partitions of the row range rather than per-row draws.
|
|
105
|
+
declared = cast("Paired", table.fields[name])
|
|
106
|
+
return partial(_from_pairing, plans[declared.relation], pairing)
|
|
90
107
|
distribution = cast("Distribution", table.fields[name])
|
|
91
108
|
return partial(_from_distribution, distribution, field_stream(seed, table.db_table, name))
|
|
92
109
|
|
|
@@ -149,6 +166,11 @@ def _from_plan(plan: FanOutPlan, row: int, values: list[object]) -> object:
|
|
|
149
166
|
return plan.key_for(row)
|
|
150
167
|
|
|
151
168
|
|
|
169
|
+
def _from_pairing(plan: FanOutPlan, pairing: PairedPlan, row: int, values: list[object]) -> object:
|
|
170
|
+
position, _size = plan.group_position(row)
|
|
171
|
+
return pairing.partner_for(plan.group_of(row), position)
|
|
172
|
+
|
|
173
|
+
|
|
152
174
|
def _from_derivation(
|
|
153
175
|
derivation: Derivation,
|
|
154
176
|
stream: int,
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
"""Key strategies whose keys cannot collide with rows already in the table."""
|
|
2
|
+
|
|
3
|
+
from __future__ import annotations
|
|
4
|
+
|
|
5
|
+
from typing import Protocol, runtime_checkable
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
@runtime_checkable
|
|
9
|
+
class Disjoint(Protocol):
|
|
10
|
+
"""A key strategy that cannot produce a key some existing row already holds.
|
|
11
|
+
|
|
12
|
+
The opt-in protocol behind one refusal, and the refusal is worth stating
|
|
13
|
+
before the protocol is. Building into a table that already has rows is
|
|
14
|
+
normally refused, because this package assigns keys from 1 every time and a
|
|
15
|
+
second build collides on the primary key -- an ``IntegrityError`` from
|
|
16
|
+
inside ``COPY`` naming an index, which tells a reader nothing.
|
|
17
|
+
|
|
18
|
+
**That reasoning is about integer keys, and the refusal was not.** A
|
|
19
|
+
:class:`~django_data_shape.keys.uuid_keys.UuidKeys` table derives a
|
|
20
|
+
128-bit digest per row and cannot collide with anything a caller's factory
|
|
21
|
+
wrote, so the refusal blocked the hybrid the documentation advertises --
|
|
22
|
+
parents made by your own code, children made by this package -- for exactly
|
|
23
|
+
the schemas where UUID keys are the norm.
|
|
24
|
+
|
|
25
|
+
A strategy that does not implement this is read as *not* disjoint, which is
|
|
26
|
+
the safe direction and the reason it is opt-in rather than a flag with a
|
|
27
|
+
default. :class:`~django_data_shape.keys.key_function.KeyFunction` is the
|
|
28
|
+
case that decides it: the caller's own function could return anything, this
|
|
29
|
+
package cannot read it, and guessing "probably fine" would trade a clear
|
|
30
|
+
refusal for a load that dies partway through.
|
|
31
|
+
|
|
32
|
+
``is_disjoint_from_existing_rows`` returns a bool rather than the protocol
|
|
33
|
+
being a bare marker, for the reason
|
|
34
|
+
:class:`~django_data_shape.distributions.distinct.Distinct` does: the answer
|
|
35
|
+
can be a property of the parameters and not only of the class.
|
|
36
|
+
|
|
37
|
+
**It says nothing about anything but keys.** A unique constraint on some
|
|
38
|
+
other column can still collide with a row that was already there, and a
|
|
39
|
+
business invariant can still be broken by rows this package did not write.
|
|
40
|
+
Both are checked after the load, against the table as it then stands, which
|
|
41
|
+
is the reading that stays true either way.
|
|
42
|
+
"""
|
|
43
|
+
|
|
44
|
+
def is_disjoint_from_existing_rows(self) -> bool: ...
|