model2data 1.14.0__tar.gz → 1.16.0__tar.gz

{model2data-1.14.0/model2data.egg-info → model2data-1.16.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: model2data
-Version: 1.14.0
+Version: 1.16.0
 Summary: Turn a data model into realistic synthetic data, a runnable dbt project and metrics with known values
 Author: JB Analytica
 License-Expression: MIT
@@ -89,9 +89,12 @@ emails, not `Lorem ipsum`. Type a column with any Faker provider (`billing_count
 come from one country. → [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md)
 
 **Foreign keys that resolve.** Tables are generated in dependency order and every foreign key
-points at a parent row that exists — onto a primary key or a unique one. Composite primary and
-unique keys hold, and a column that repeats a parent's value through a foreign key
-(`orders.customer_email`) holds the email of the order's own customer.
+points at a parent row that exists — onto a primary key or a unique one — and
+`after: customers.created_at` keeps an order on or after the sign-up of the customer it points
+at (`validate` suggests it where a date could fall before its parent's). Integer primary keys
+number the rows 1..N, in order. Composite primary and unique keys hold, and a column that repeats a
+parent's value through a foreign key (`orders.customer_email`) holds the email of the order's
+own customer.
 → [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md)
 
 **A dbt project that runs.** Seeds, staging models that `ref()` them, `not_null`, `unique`,
@@ -115,8 +118,7 @@ byte-identical. → [The determinism promise](https://github.com/JB-Analytica/mo
 
 **When things happen.** `--business-hours`, `--growth` and `--seasonality` shape every timestamp
 (weekdays and working hours, a trend, a Q4 peak), or one column at a time. `updated_at` never
-lands before `created_at`, `after` orders any two columns, and `when` gives `shipped_at` a value
-on shipped and delivered orders only. → [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md)
+lands before `created_at`, `after` orders any two columns, or a column after a parent's, and `when` gives `shipped_at` a value on shipped and delivered orders only. → [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md)
 
 **How the data is spread.** `--skew` lets a few customers place most of the orders; `weights`,
 `true_rate`, `null_rate` and `distinct` shape a column; `distribution` draws a number from a

{model2data-1.14.0 → model2data-1.16.0}/README.md

@@ -42,9 +42,12 @@ emails, not `Lorem ipsum`. Type a column with any Faker provider (`billing_count
 come from one country. → [The model file](docs/model-file.md)
 
 **Foreign keys that resolve.** Tables are generated in dependency order and every foreign key
-points at a parent row that exists — onto a primary key or a unique one. Composite primary and
-unique keys hold, and a column that repeats a parent's value through a foreign key
-(`orders.customer_email`) holds the email of the order's own customer.
+points at a parent row that exists — onto a primary key or a unique one — and
+`after: customers.created_at` keeps an order on or after the sign-up of the customer it points
+at (`validate` suggests it where a date could fall before its parent's). Integer primary keys
+number the rows 1..N, in order. Composite primary and unique keys hold, and a column that repeats a
+parent's value through a foreign key (`orders.customer_email`) holds the email of the order's
+own customer.
 → [The model file](docs/model-file.md)
 
 **A dbt project that runs.** Seeds, staging models that `ref()` them, `not_null`, `unique`,
@@ -68,8 +71,7 @@ byte-identical. → [The determinism promise](docs/generating.md#the-determinism
 
 **When things happen.** `--business-hours`, `--growth` and `--seasonality` shape every timestamp
 (weekdays and working hours, a trend, a Q4 peak), or one column at a time. `updated_at` never
-lands before `created_at`, `after` orders any two columns, and `when` gives `shipped_at` a value
-on shipped and delivered orders only. → [When things happen](docs/time-shapes.md)
+lands before `created_at`, `after` orders any two columns, or a column after a parent's, and `when` gives `shipped_at` a value on shipped and delivered orders only. → [When things happen](docs/time-shapes.md)
 
 **How the data is spread.** `--skew` lets a few customers place most of the orders; `weights`,
 `true_rate`, `null_rate` and `distinct` shape a column; `distribution` draws a number from a

{model2data-1.14.0 → model2data-1.16.0}/README_PYPI.md

@@ -42,9 +42,12 @@ emails, not `Lorem ipsum`. Type a column with any Faker provider (`billing_count
 come from one country. → [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md)
 
 **Foreign keys that resolve.** Tables are generated in dependency order and every foreign key
-points at a parent row that exists — onto a primary key or a unique one. Composite primary and
-unique keys hold, and a column that repeats a parent's value through a foreign key
-(`orders.customer_email`) holds the email of the order's own customer.
+points at a parent row that exists — onto a primary key or a unique one — and
+`after: customers.created_at` keeps an order on or after the sign-up of the customer it points
+at (`validate` suggests it where a date could fall before its parent's). Integer primary keys
+number the rows 1..N, in order. Composite primary and unique keys hold, and a column that repeats a
+parent's value through a foreign key (`orders.customer_email`) holds the email of the order's
+own customer.
 → [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md)
 
 **A dbt project that runs.** Seeds, staging models that `ref()` them, `not_null`, `unique`,
@@ -68,8 +71,7 @@ byte-identical. → [The determinism promise](https://github.com/JB-Analytica/mo
 
 **When things happen.** `--business-hours`, `--growth` and `--seasonality` shape every timestamp
 (weekdays and working hours, a trend, a Q4 peak), or one column at a time. `updated_at` never
-lands before `created_at`, `after` orders any two columns, and `when` gives `shipped_at` a value
-on shipped and delivered orders only. → [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md)
+lands before `created_at`, `after` orders any two columns, or a column after a parent's, and `when` gives `shipped_at` a value on shipped and delivered orders only. → [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md)
 
 **How the data is spread.** `--skew` lets a few customers place most of the orders; `weights`,
 `true_rate`, `null_rate` and `distinct` shape a column; `distribution` draws a number from a

{model2data-1.14.0 → model2data-1.16.0}/model2data/cli.py

@@ -29,6 +29,7 @@ from model2data.defects import (
 )
 from model2data.generate.core import (
     generate_data_from_dbml,
+    get_capped_tables,
     get_cyclic_tables,
     get_unresolved_composite_keys,
 )
@@ -184,7 +185,7 @@ app = typer.Typer(
     cls=_GenerateByDefault,
     help=(
         "model2data: Generate analytics-ready datasets from a data model.\n\n"
-        "Given a model -- a .model2data.yml document (spec 0.4.0), the same as JSON,\n"
+        "Given a model -- a .model2data.yml document (spec 0.5.0), the same as JSON,\n"
         "or a DBML file -- this tool produces:\n"
         "• Synthetic but realistic data\n"
         "• A runnable dbt project scaffold\n"
@@ -401,8 +402,9 @@ def main(
         "warn",
         "--hint-tests",
         help=(
-            "Write the model's generation hints as dbt tests (min/max range, after, "
-            "null_rate, distinct, when, grain) at this severity: error, warn or off. They "
+            "Write the model's generation hints as dbt tests (min/max range, after "
+            "(also a parent's column), null_rate, distinct, when, grain) at this severity: "
+            "error, warn or off. They "
             "describe intent, so they warn by default rather than break a first dbt "
             "build on real data."
         ),
@@ -772,6 +774,7 @@ def main(
     cyclic_tables = get_cyclic_tables()
     unresolved_keys = get_unresolved_composite_keys()
     duplicate_unique = get_duplicate_unique_columns()
+    capped_tables = get_capped_tables()
 
     typer.echo("\n📊 Summary")
     typer.echo(f"  Tables generated:        {len(generated_tables)}")
@@ -810,6 +813,13 @@ def main(
         )
         for label in duplicate_unique:
             typer.echo(f"    - {label}")
+    if capped_tables:
+        typer.echo(
+            "  ⚠️  Tables generated with fewer rows than asked, so a unique reference "
+            "keeps one row per parent value:"
+        )
+        for label in capped_tables:
+            typer.echo(f"    - {label}")
 
     if metrics_count is not None:
         typer.echo(
@@ -837,12 +847,13 @@ def main(
 
 
 def _spec_of(file: Path) -> str:
-    """The spec a conforming model is written against: 0.4.0, 0.3.0, or 0.2.0 (DBML without
-    `when` included)."""
+    """The spec a conforming model is written against: 0.5.0, 0.4.0, 0.3.0, or 0.2.0 (DBML
+    without `when` or a 0.5.0 `after` included)."""
     version = str(load(file).version)
-    return (
-        "0.4.0" if version.startswith("0.4") else "0.3.0" if version.startswith("0.3") else "0.2.0"
-    )
+    for minor in ("0.5", "0.4", "0.3"):
+        if version.startswith(minor):
+            return f"{minor}.0"
+    return "0.2.0"
 
 
 def _print_defects(report: DefectsReport) -> None:
@@ -934,11 +945,12 @@ def validate_command(
         ),
     ),
 ):
-    """Check that models conform to spec 0.4.0 (or 0.2.x, 0.3.x), printing every issue with its path.
+    """Check that models conform to spec 0.5.0 (or 0.2.x-0.4.x), printing every issue with its path.
 
     Takes one or more files and/or --glob patterns. A metrics file (*.metrics.yml) is checked
     against metrics spec 0.1.0 and against its model. Exits 1 when any file has an error;
-    warnings are printed, and the file conforms.
+    warnings are printed, and the file conforms. A warning that a child's date `can fall
+    before` its parent's names the `after: <table>.<column>` to add.
     """
     if output_format not in ("text", "github"):
         raise typer.BadParameter("must be 'text' or 'github'", param_hint="--format")
@@ -1029,7 +1041,8 @@ def convert_command(
     force: bool = typer.Option(False, "--force", help="Overwrite the output file if it exists."),
 ):
     """Convert a model -- DBML, typically -- to a .model2data.yml document (spec 0.2.0, or
-    0.3.0 when it has defects, 0.4.0 when it has `when`)."""
+    0.3.0 when it has defects, 0.4.0 when it has `when`, 0.5.0 when an `after` is a list or
+    names a parent's column)."""
     try:
         model = load(file)
     except ModelError as error:

{model2data-1.14.0 → model2data-1.16.0}/model2data/dbt/hint_tests.py

@@ -12,11 +12,17 @@ What becomes a test, and what does not:
 |---------------------------|----------------------------------------|-----------|
 | `generate.min` / `max`    | `model2data_between`                   | none      |
 | `generate.after: other`   | `model2data_not_before`                | none      |
+| `generate.after: table.column` | `model2data_not_before_parent`    | none      |
 | `generate.null_rate`      | `model2data_max_null_share`            | yes       |
 | `generate.distinct: n`    | `model2data_max_distinct`              | none      |
 | `generate.when`           | `model2data_when`                      | none      |
 | table `grain`             | `model2data_unique_combination`        | none      |
 
+An `after` listing several columns writes one test for each. A parent's column
+(`after: customers.created_at`, see `model2data.generate.parents`) is tested
+through the foreign key that reaches it, so that test needs the model's
+references: a caller passing bare tables passes `refs` too.
+
 A column with `when` and `null_rate` has its null share tested among the rows
 `when` matches (`model2data_when_max_null_share`), since only those rows can
 hold a value, and its `model2data_when` test lets those rows be null.
@@ -28,8 +34,9 @@ constraint a row can break, so they produce no test.
 
 The generic tests live in `macros/model2data_hint_tests.sql` of the generated
 project (see `write_hint_macros`), and the `when` ones in
-`macros/model2data_when_tests.sql`, written only for a model that has one, so
-a project without `when` is what it was: plain SQL, no dbt package.
+`macros/model2data_when_tests.sql` and the parent one in
+`macros/model2data_parent_tests.sql`, each written only for a model that has
+one, so a project without either is what it was: plain SQL, no dbt package.
 """
 
 from __future__ import annotations
@@ -41,6 +48,9 @@ from pathlib import Path
 from typing import Any, Literal, Optional, Union
 
 from model2data.generate import kinds
+from model2data.generate.core import parent_rules_for
+from model2data.generate.parents import ParentRule
+from model2data.generate.timeline import same_row_after
 from model2data.model.engine import EngineInputs, to_engine
 from model2data.model.types import Model
 from model2data.parse.dbml import ColumnDef, TableDef
@@ -53,6 +63,8 @@ MACROS_FILE = "model2data_hint_tests.sql"
 _MACROS_SOURCE = Path(__file__).parent / "templates" / "hint_macros" / MACROS_FILE
 WHEN_MACROS_FILE = "model2data_when_tests.sql"
 _WHEN_MACROS_SOURCE = Path(__file__).parent / "templates" / "hint_macros" / WHEN_MACROS_FILE
+PARENT_MACROS_FILE = "model2data_parent_tests.sql"
+_PARENT_MACROS_SOURCE = Path(__file__).parent / "templates" / "hint_macros" / PARENT_MACROS_FILE
 
 
 @dataclass(frozen=True)
@@ -64,6 +76,8 @@ class HintTest:
     test's name, `arguments` its parameters as dbt's `arguments:` block, and
     `hint` the hint it comes from (`min`, `max`, `after`, `null_rate`,
     `distinct`, `when`, `grain`). `min` and `max` of one column share one test.
+    `parent` is the `(table, column)` a test looks rows up in: the parent's key
+    for `model2data_not_before_parent`, None for every other test.
     """
 
     table: str
@@ -71,6 +85,7 @@ class HintTest:
     test: str
     arguments: dict[str, Any] = field(default_factory=dict)
     hint: str = ""
+    parent: Optional[tuple[str, str]] = None
 
     def to_dbt(self, severity: str = "warn") -> dict[str, dict[str, Any]]:
         """The entry for a `tests:` list: `{name: {arguments: ..., config: {severity: ...}}}`."""
@@ -83,12 +98,16 @@ def hint_tests_for(
     model_or_tables: Union[Model, EngineInputs, Mapping[str, TableDef]],
     *,
     tolerance: float = DEFAULT_TOLERANCE,
+    refs: Optional[list[dict]] = None,
 ) -> list[HintTest]:
     """The tests implied by the hints of a model, in table then column order.
 
     Takes a `Model`, the `EngineInputs` of one, or the `tables` dict (as
     `to_engine(model).tables`, or the renamed tables the CLI hands
     `generate_dbt_yml`). Table-level tests follow the table's column tests.
+    With `tables`, `refs` are the references as `to_engine` gives them; without
+    them no test of a parent's column is written (a `Model` or `EngineInputs`
+    brings its own).
 
     `tolerance` is the absolute slack of the statistical test (the null share
     may be `null_rate + tolerance`); hard constraints take none. Severity is
@@ -97,18 +116,25 @@ def hint_tests_for(
     if tolerance < 0:
         raise ValueError("tolerance must not be negative")
     if isinstance(model_or_tables, Model):
-        tables: Mapping[str, TableDef] = to_engine(model_or_tables).tables
+        inputs = to_engine(model_or_tables)
+        tables: Mapping[str, TableDef] = inputs.tables
+        refs = inputs.refs
     elif isinstance(model_or_tables, EngineInputs):
         tables = model_or_tables.tables
+        refs = model_or_tables.refs
     else:
         tables = model_or_tables
+    rules = parent_rules_for(dict(tables), refs) if refs else {}
 
     found: list[HintTest] = []
     for key, table in tables.items():
         by_name = {column.name: column for column in table.columns}
         names = set(by_name)
+        parents = {rule.column: rule for rule in rules.get(key, [])}
         for column in table.columns:
             found.extend(_column_tests(key, column, by_name, tolerance))
+            if column.name in parents:
+                found.extend(_parent_tests(key, parents[column.name]))
         grain = (table.note or {}).get("grain")
         if grain and all(name in names for name in grain):
             found.append(
@@ -146,8 +172,7 @@ def _column_tests(
                 )
             )
 
-    after = note.get("after")
-    if isinstance(after, str) and after in kinds_of and after != column.name:
+    for after in [a for a in same_row_after(note) if a in kinds_of and a != column.name]:
         arguments: dict[str, Any] = {"other": after}
         # The generator compares a date with a timestamp at day granularity.
         if (
@@ -202,6 +227,31 @@ def _column_tests(
     return tests
 
 
+def _parent_tests(table: str, rule: ParentRule) -> list[HintTest]:
+    """One `model2data_not_before_parent` test per parent column the `after` names."""
+    tests = []
+    for link in rule.links:
+        arguments: dict[str, Any] = {
+            "foreign_key": link.fk,
+            "to": f"ref('stg_{link.parent_table}')",
+            "field": link.parent_key,
+            "parent_column": link.parent_column,
+        }
+        if link.by_day:
+            arguments["granularity"] = "day"
+        tests.append(
+            HintTest(
+                table,
+                rule.column,
+                "model2data_not_before_parent",
+                arguments,
+                "after",
+                parent=(link.parent_table, link.parent_key),
+            )
+        )
+    return tests
+
+
 def _when_conditions(when: Any, kinds_of: dict[str, ColumnDef]) -> Optional[dict[str, list]]:
     """A `when` hint as the test's `conditions`: an enum's values as the member text it holds."""
     if not isinstance(when, dict) or not when or not all(name in kinds_of for name in when):
@@ -212,13 +262,14 @@ def _when_conditions(when: Any, kinds_of: dict[str, ColumnDef]) -> Optional[dict
     }
 
 
-def write_hint_macros(dest: Path, *, when: bool = False) -> Path:
+def write_hint_macros(dest: Path, *, when: bool = False, parents: bool = False) -> Path:
     """Write the generic tests into `dest/macros/`, and return the file's path.
 
     A project whose YAML holds `hint_tests_for` tests needs this file next to
     it; `generate_dbt_yml` writes it itself when it emits any. With `when`, the
     file of the `when` tests is written beside it (`WHEN_MACROS_FILE`), which a
-    project holding a `model2data_when` test needs too.
+    project holding a `model2data_when` test needs too; with `parents`, the
+    file of `model2data_not_before_parent` (`PARENT_MACROS_FILE`).
     """
     macros = dest / "macros"
     macros.mkdir(parents=True, exist_ok=True)
@@ -226,4 +277,6 @@ def write_hint_macros(dest: Path, *, when: bool = False) -> Path:
     target.write_text(_MACROS_SOURCE.read_text())
     if when:
         (macros / WHEN_MACROS_FILE).write_text(_WHEN_MACROS_SOURCE.read_text())
+    if parents:
+        (macros / PARENT_MACROS_FILE).write_text(_PARENT_MACROS_SOURCE.read_text())
     return target

model2data-1.16.0/model2data/dbt/templates/hint_macros/model2data_parent_tests.sql

@@ -0,0 +1,37 @@
+{#-
+  The generic test model2data writes for a cross-table `after` (`after:
+  customers.created_at` on orders.order_date): a row is not dated before the
+  parent row its foreign key points at. Written beside
+  model2data_hint_tests.sql, and only by a model that has such a hint. Plain
+  ANSI SQL, runs on DuckDB and Postgres. Returns the rows that break it.
+  `column_name` arrives already quoted by the schema YAML; `foreign_key`,
+  `field` (the parent's key) and `parent_column` (the column `after` names)
+  are raw names and are quoted here. A row whose foreign key is null, or points
+  at no parent row, is left to the not_null and relationships tests, and a key
+  several parent rows share (left to the unique test) counts its earliest one.
+  `granularity='day'` compares a date with a timestamp by day.
+-#}
+
+{% test model2data_not_before_parent(model, column_name, foreign_key, to, field, parent_column, granularity=none) %}
+{%- set mine = 'child.' ~ column_name -%}
+{%- set theirs = 'parent.parent_value' -%}
+{%- if granularity == 'day' -%}
+  {%- set mine = 'cast(' ~ mine ~ ' as date)' -%}
+  {%- set theirs = 'cast(' ~ theirs ~ ' as date)' -%}
+{%- endif %}
+select
+  child.{{ column_name }} as value,
+  parent.parent_value
+from {{ model }} as child
+join (
+  select
+    {{ adapter.quote(field) }} as parent_key,
+    min({{ adapter.quote(parent_column) }}) as parent_value
+  from {{ to }}
+  where {{ adapter.quote(parent_column) }} is not null
+  group by {{ adapter.quote(field) }}
+) as parent
+  on child.{{ adapter.quote(foreign_key) }} = parent.parent_key
+where child.{{ column_name }} is not null
+  and {{ mine }} < {{ theirs }}
+{% endtest %}

{model2data-1.14.0 → model2data-1.16.0}/model2data/dbt/tests.py

@@ -66,10 +66,12 @@ def generate_dbt_yml(
     staging_path = dest / "models" / "staging"
     staging_path.mkdir(parents=True, exist_ok=True)
 
-    hinted = _hinted(tables, hint_tests, test_tolerance)
+    hinted = _hinted(tables, hint_tests, test_tolerance, refs)
     if hinted:
-        when = any(test.hint == "when" for tests in hinted.values() for test in tests)
-        write_hint_macros(dest, when=when)
+        names = {test.test for tests in hinted.values() for test in tests}
+        write_hint_macros(
+            dest, when="model2data_when" in names, parents="model2data_not_before_parent" in names
+        )
     if any(_history_entries(table) for table in tables.values()):
         write_history_macros(dest)
     fk_map = _fk_map(tables, refs)
@@ -114,14 +116,14 @@ def generate_dbt_yml(
 
 
 def _hinted(
-    tables: dict, hint_tests: str, test_tolerance: float
+    tables: dict, hint_tests: str, test_tolerance: float, refs: list[dict]
 ) -> dict[tuple[str, Optional[str]], list[HintTest]]:
     """The hint tests by (table, column), column None for a table-level one; none for `off`."""
     if hint_tests not in ("error", "warn", "off"):
         raise ValueError(f"hint_tests must be error, warn or off, not {hint_tests!r}")
     hinted: dict[tuple[str, Optional[str]], list[HintTest]] = defaultdict(list)
     if hint_tests != "off":
-        for hint_test in hint_tests_for(tables, tolerance=test_tolerance):
+        for hint_test in hint_tests_for(tables, tolerance=test_tolerance, refs=refs):
             hinted[(hint_test.table, hint_test.column)].append(hint_test)
     return hinted
 
@@ -193,7 +195,8 @@ class DbtTest:
     `relationships`, a `model2data_*` hint test) or `unique_combination` for the
     singular test of a composite key. `arguments` are the test's parameters,
     `severity` is `error`, or the hint tests' severity, and `parent` is the
-    `(table, column)` a `relationships` test looks values up in.
+    `(table, column)` a `relationships` test looks values up in, or the parent
+    key a `model2data_not_before_parent` test joins on.
     """
 
     name: str
@@ -218,7 +221,7 @@ def dbt_tests(
     name dbt gives the test in the generated project. Unit tests are not data
     tests and are not listed.
     """
-    hinted = _hinted(tables, hint_tests, test_tolerance)
+    hinted = _hinted(tables, hint_tests, test_tolerance, refs)
     fk_map = _fk_map(tables, refs)
     found: list[DbtTest] = []
     for table in tables.values():
@@ -226,6 +229,9 @@ def dbt_tests(
         for col in table.columns:
             column_ref = _dbt_column_ref(col.name)
             parents = iter(fk_map.get((table.name, col.name), []))
+            hint_parents = iter(
+                t.parent for t in hinted.get((table.name, col.name), []) if t.parent is not None
+            )
             for entry in _column_entries(table, col, fk_map, hinted, hint_tests):
                 test_type, body = _entry_parts(entry)
                 arguments = dict(body.get("arguments") or {})
@@ -233,6 +239,8 @@ def dbt_tests(
                 if test_type == "relationships":
                     ref = next(parents)
                     parent = (ref["target_table"], ref["target_column"])
+                elif test_type == "model2data_not_before_parent":
+                    parent = next(hint_parents)
                 found.append(
                     DbtTest(
                         name=generic_test_name(

{model2data-1.14.0 → model2data-1.16.0}/model2data/defects/apply.py

@@ -67,8 +67,10 @@ from model2data.dbt.tests import DbtTest, dbt_tests
 from model2data.defects.checks import Columns, as_seeded, failing, fails
 from model2data.defects.report import AppliedDefect, DefectsReport, ExpectedFailure
 from model2data.generate import kinds
+from model2data.generate.core import parent_rules_for
 from model2data.generate.days import DayResult, TableDay
 from model2data.generate.history import IS_CURRENT, VALID_FROM, VALID_TO, history_key, primary_key
+from model2data.generate.parents import ParentRule, floor_on_day, moments
 from model2data.model.engine import EngineInputs, to_engine
 from model2data.model.types import Defect, Model
 from model2data.parse.dbml import ColumnDef, TableDef
@@ -111,8 +113,11 @@ class _Table:
         refs: list[dict],
         results: Optional[list[DayResult]],
         incremental: Any,
+        rules: Sequence[ParentRule] = (),
     ):
         self.key = key
+        # The columns whose `after` names a parent's column (see generate.parents).
+        self.rules = list(rules)
         self.table = table
         self.frames = frames
         self.frame = frames[key]
@@ -533,7 +538,7 @@ def _late_arriving(table: _Table, defect: Defect, rng: random.Random, count: int
     ]
     temporal = [c for c in table.columns if table.is_temporal(c)]
     candidates = table.free(candidates, temporal)
-    chosen = _choose(rng, candidates, count, set())
+    chosen = _choose(rng, candidates, count, _before_parents_once_late(table, candidates, name))
     patches: list[_Patch] = []
     # Earliest day first: a row moved back lowers what the loader has seen on later days.
     for position in sorted(chosen, key=lambda p: (table.inserted_day[p], p)):
@@ -634,6 +639,40 @@ def _event_column(table: _Table) -> Optional[str]:
     return next((c for c in table.columns if table.is_temporal(c)), None)
 
 
+def _before_parents_once_late(table: _Table, candidates: list[int], name: str) -> set[int]:
+    """The candidates moving back would date before a parent row (see generate.parents).
+
+    A late-arriving row is moved back past the previous load's cutoff, every date of
+    it; a date with a cross-table `after` can then fall before its parent, which the
+    dbt test `model2data_not_before_parent` reports. Rows that stay clear of their
+    parents are taken first, so the defect breaks the incremental model it is meant
+    to and no test besides.
+    """
+    if not table.rules or not candidates:
+        return set()
+    checks = [
+        (
+            floor_on_day(table.frame, rule, table.frames).tolist(),
+            moments(table.frame[rule.column], rule.kind).tolist(),
+        )
+        for rule in table.rules
+    ]
+    values = table.frame[name].tolist()
+    cutoffs: dict[int, Any] = {}
+    risky = set()
+    for position in candidates:
+        day = table.inserted_day[position]
+        if day not in cutoffs:
+            cutoffs[day] = table.loaded_max(day - 1, name)
+        cutoff, moment, back = cutoffs[day], _moment(values[position]), 1
+        while cutoff is not None and moment - timedelta(days=back) >= cutoff:
+            back += 1
+        # A null foreign key or date gives NaT, which compares false.
+        if any(own[position] - pd.Timedelta(days=back) < floor[position] for floor, own in checks):
+            risky.add(position)
+    return risky
+
+
 def _moment(value: Any) -> datetime:
     return pd.Timestamp(value).to_pydatetime()
 
@@ -818,6 +857,7 @@ def break_tables(
 
     applied: Applied = []
     tables: dict[str, _Table] = {}
+    rules = parent_rules_for(inputs.tables, inputs.refs)
     for key, entries in defects.items():
         if key not in inputs.tables:
             raise ValueError(f"defects name the table {key!r}, which is not in the model")
@@ -840,6 +880,7 @@ def break_tables(
                     inputs.refs,
                     results,
                     inputs.incremental.get(target),
+                    rules.get(target, []),
                 )
             table = tables[target]
             rng = random.Random(defect_stream_seed(base_seed, key, defect))

{model2data-1.14.0 → model2data-1.16.0}/model2data/defects/checks.py

@@ -14,6 +14,9 @@ SQL reads:
   and `model2data_max_distinct`: as their macros in
   `dbt/templates/hint_macros/model2data_hint_tests.sql`, nulls skipped where the
   SQL skips them.
+- `model2data_not_before_parent`: as its macro in
+  `dbt/templates/hint_macros/model2data_parent_tests.sql`: a row dated before
+  the earliest parent row its foreign key matches, by day with `granularity: day`.
 - `model2data_when` and `model2data_when_max_null_share`: as their macros in
   `dbt/templates/hint_macros/model2data_when_tests.sql`, a row matching when
   each named column's text is one of its values (a boolean as `true`/`false`
@@ -338,6 +341,39 @@ def _not_before(test: DbtTest, frame: pd.DataFrame, seeded: Mapping, cache: Colu
     return bool((mine < theirs).any())
 
 
+def _not_before_parent(test: DbtTest, frame: pd.DataFrame, seeded: Mapping, cache: Columns) -> bool:
+    assert test.parent is not None  # `dbt_tests` names the parent of every such test
+    parent = seeded.get(test.parent[0])
+    if parent is None:
+        return False
+    arguments = test.arguments
+    mine = pd.DataFrame(
+        {
+            "key": cache.numbers(frame, arguments["foreign_key"]),
+            "moment": cache.values(frame, str(test.column)),
+        }
+    ).dropna()
+    theirs = pd.DataFrame(
+        {
+            "key": cache.numbers(parent, arguments["field"]),
+            "parent_moment": cache.values(parent, arguments["parent_column"]),
+        }
+    ).dropna()
+    if mine.empty or theirs.empty:
+        return False
+    # A key several parent rows share counts its earliest, as the SQL's `min` does.
+    theirs["parent_moment"] = pd.to_datetime(theirs["parent_moment"], format="mixed")
+    earliest = theirs.groupby("key", sort=False)["parent_moment"].min().reset_index()
+    pairs = mine.merge(earliest, on="key")
+    if pairs.empty:
+        return False
+    child = pd.to_datetime(pairs["moment"], format="mixed")
+    parents = pairs["parent_moment"]
+    if arguments.get("granularity") == "day":
+        child, parents = child.dt.normalize(), parents.dt.normalize()
+    return bool((child < parents).any())
+
+
 def _max_null_share(test: DbtTest, frame: pd.DataFrame, seeded: Mapping, cache: Columns) -> bool:
     if not len(frame):
         return False
@@ -433,6 +469,7 @@ _CHECKS: dict[str, Callable[[DbtTest, pd.DataFrame, Mapping, Columns], bool]] =
     "relationships": _relationships,
     "model2data_between": _between,
     "model2data_not_before": _not_before,
+    "model2data_not_before_parent": _not_before_parent,
     "model2data_max_null_share": _max_null_share,
     "model2data_max_distinct": _max_distinct,
     "model2data_when": _when,