model2data 1.10.4__tar.gz → 1.11.0__tar.gz

{model2data-1.10.4/model2data.egg-info → model2data-1.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: model2data
-Version: 1.10.4
+Version: 1.11.0
 Summary: Generate analytics-ready datasets from DBML models
 Author: JB Analytica
 License-Expression: MIT
@@ -132,7 +132,7 @@ access required.
 ## The model file
 
 A model is one YAML document, `<name>.model2data.yml` — or the same document as JSON. Its format
-is [spec 0.3.0](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md), with a JSON Schema your editor can autocomplete and
+is [spec 0.4.0](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md), with a JSON Schema your editor can autocomplete and
 check against:
 
 ```yaml
@@ -292,6 +292,7 @@ dbt package, so `dbt build` still works offline):
 | `generate: {after: other}` | `model2data_not_before` (where both are not null) | none |
 | `generate: {null_rate}` | `model2data_max_null_share`: nulls at most `null_rate` + tolerance | `--test-tolerance`, default 0.1 |
 | `generate: {distinct: n}` | `model2data_max_distinct`: at most `n` distinct values | none |
+| `generate: {when}` | `model2data_when`: null exactly where the condition does not hold (with `null_rate`, only there; its null share is then tested among the matching rows) | none |
 | `grain` on a table | `model2data_unique_combination` | none |
 | an enum-typed column | `accepted_values` (always written) | none |
 

{model2data-1.10.4 → model2data-1.11.0}/README.md

@@ -16,6 +16,13 @@ relationship-preserving synthetic data
 DuckDB or Postgres profile. No sample data to hunt down, no dbt boilerplate to hand-write, no
 production data to risk exposing.
 
+Driving model2data from a coding agent? Have it start here; the instructions ship with the
+installed version and pick a topic from the current directory:
+
+```bash
+uvx model2data@latest guide
+```
+
 A working analytics stack — real (synthetic) data, tested dbt models, queryable in
 DuckDB — from a schema file, in seconds:
 
@@ -123,7 +130,7 @@ flowchart LR
 ## The model file
 
 A model is one YAML document, `<name>.model2data.yml` — or the same document as JSON. Its
-format is [spec 0.3.0](model2data/spec/README.md), with a JSON Schema
+format is [spec 0.4.0](model2data/spec/README.md), with a JSON Schema
 ([`model.schema.json`](model2data/spec/model.schema.json)) your editor can autocomplete and check
 against:
 
@@ -222,14 +229,14 @@ jobs:
 ```
 
 Inputs: `files` (glob, default `**/*.model2data.yml`), `version` (default: the release the tag
-points at, e.g. `1.10.4`; a specifier such as `>=1.10,<2` also works), `python-version` (3.12).
+points at, e.g. `1.11.0`; a specifier such as `>=1.10,<2` also works), `python-version` (3.12).
 
 **pre-commit**
 
 ```yaml
 repos:
   - repo: https://github.com/JB-Analytica/model2data
-    rev: v1.10.4
+    rev: v1.11.0
     hooks:
       - id: model2data-validate
 ```
@@ -243,7 +250,7 @@ model2data:
     - if: $CI_PIPELINE_SOURCE == "merge_request_event"
       changes: ["**/*.model2data.yml"]
   script:
-    - pip install model2data==1.10.4
+    - pip install model2data==1.11.0
     - model2data validate --glob "**/*.model2data.yml"
 ```
 
@@ -394,6 +401,24 @@ shipped_at:
   generate: {after: ordered_at}
 ```
 
+A column that only has a value in some states says which with `when` (spec 0.4.0: the model says
+`model2data: 0.4.0`): it holds a value on
+exactly the rows whose named column holds one of the listed values, and is null on every other
+row — no `pending` order with a `shipped_at`, no `cancelled` subscription without a
+`cancelled_at`:
+
+```yaml
+shipped_at:
+  type: timestamp
+  generate: {after: ordered_at, when: {status: [shipped, delivered]}}
+```
+
+The named column is another column of the table (an enum, boolean, number or text column; several
+named must all match), and the column carrying `when` is nullable. On the matching rows it is
+never null, unless it has a `null_rate`, which then counts only those rows. On days after the
+first (`--days`), a row that moves into a listed status gets its value that day and a row that
+moves out loses it. Every other column comes out exactly as it would without the hint.
+
 The flags above (or `run.shape` in the model) shape every date and timestamp column the same
 way, run-wide. `business_hours`, `growth`, and `seasonality` column hints override that for one
 column at a time — the whole point being a run can be uniform everywhere except the one column
@@ -525,6 +550,7 @@ dbt package, so `dbt build` still works offline):
 | `generate: {after: other}` | `model2data_not_before` (where both are not null) | none |
 | `generate: {null_rate}` | `model2data_max_null_share`: nulls at most `null_rate` + tolerance | `--test-tolerance`, default 0.1 |
 | `generate: {distinct: n}` | `model2data_max_distinct`: at most `n` distinct values | none |
+| `generate: {when}` | `model2data_when`: null exactly where the condition does not hold (with `null_rate`, only there; its null share is then tested among the matching rows) | none |
 | `grain` on a table | `model2data_unique_combination` | none |
 | an enum-typed column | `accepted_values` (always written) | none |
 

{model2data-1.10.4 → model2data-1.11.0}/README_PYPI.md

@@ -85,7 +85,7 @@ access required.
 ## The model file
 
 A model is one YAML document, `<name>.model2data.yml` — or the same document as JSON. Its format
-is [spec 0.3.0](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md), with a JSON Schema your editor can autocomplete and
+is [spec 0.4.0](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md), with a JSON Schema your editor can autocomplete and
 check against:
 
 ```yaml
@@ -245,6 +245,7 @@ dbt package, so `dbt build` still works offline):
 | `generate: {after: other}` | `model2data_not_before` (where both are not null) | none |
 | `generate: {null_rate}` | `model2data_max_null_share`: nulls at most `null_rate` + tolerance | `--test-tolerance`, default 0.1 |
 | `generate: {distinct: n}` | `model2data_max_distinct`: at most `n` distinct values | none |
+| `generate: {when}` | `model2data_when`: null exactly where the condition does not hold (with `null_rate`, only there; its null share is then tested among the matching rows) | none |
 | `grain` on a table | `model2data_unique_combination` | none |
 | an enum-typed column | `accepted_values` (always written) | none |
 

{model2data-1.10.4 → model2data-1.11.0}/model2data/cli.py

@@ -2,6 +2,8 @@ import random
 import shutil
 from collections.abc import Iterable
 from datetime import date, datetime
+from enum import Enum
+from importlib import resources
 from pathlib import Path
 from typing import Any, Optional
 
@@ -163,13 +165,13 @@ app = typer.Typer(
     cls=_GenerateByDefault,
     help=(
         "model2data: Generate analytics-ready datasets from a data model.\n\n"
-        "Given a model -- a .model2data.yml document (spec 0.3.0), the same as JSON,\n"
+        "Given a model -- a .model2data.yml document (spec 0.4.0), the same as JSON,\n"
         "or a DBML file -- this tool produces:\n"
         "• Synthetic but realistic data\n"
         "• A runnable dbt project scaffold\n"
         "• dbt seeds, staging models, and profiles\n\n"
         "`model2data --file MODEL` generates; `validate` and `convert` check and\n"
-        "convert a model."
+        "convert a model. Agents: run `model2data guide` first."
     ),
     add_completion=False,
 )
@@ -369,7 +371,7 @@ def main(
         "--hint-tests",
         help=(
             "Write the model's generation hints as dbt tests (min/max range, after, "
-            "null_rate, distinct, grain) at this severity: error, warn or off. They "
+            "null_rate, distinct, when, grain) at this severity: error, warn or off. They "
             "describe intent, so they warn by default rather than break a first dbt "
             "build on real data."
         ),
@@ -735,8 +737,12 @@ def main(
 
 
 def _spec_of(file: Path) -> str:
-    """The spec a conforming model is written against: 0.3.0, or 0.2.0 (DBML included)."""
-    return "0.3.0" if str(load(file).version).startswith("0.3") else "0.2.0"
+    """The spec a conforming model is written against: 0.4.0, 0.3.0, or 0.2.0 (DBML without
+    `when` included)."""
+    version = str(load(file).version)
+    return (
+        "0.4.0" if version.startswith("0.4") else "0.3.0" if version.startswith("0.3") else "0.2.0"
+    )
 
 
 def _print_defects(report: DefectsReport) -> None:
@@ -815,7 +821,7 @@ def validate_command(
         False, "--require-files", help="Exit 1 when no model file was given or matched."
     ),
 ):
-    """Check that models conform to spec 0.3.0 (or 0.2.x), printing every issue with its path.
+    """Check that models conform to spec 0.4.0 (or 0.2.x, 0.3.x), printing every issue with its path.
 
     Takes one or more files and/or --glob patterns. Exits 1 when any file has an error;
     warnings are printed, and the model conforms.
@@ -874,7 +880,7 @@ def convert_command(
     force: bool = typer.Option(False, "--force", help="Overwrite the output file if it exists."),
 ):
     """Convert a model -- DBML, typically -- to a .model2data.yml document (spec 0.2.0, or
-    0.3.0 when it has defects)."""
+    0.3.0 when it has defects, 0.4.0 when it has `when`)."""
     try:
         model = load(file)
     except ModelError as error:
@@ -891,3 +897,47 @@ def convert_command(
         raise typer.Exit(1)
     output.write_text(text, encoding="utf-8")
     typer.echo(f"✅ Wrote {output}")
+
+
+class GuideTopic(str, Enum):
+    setup = "setup"
+    triage = "triage"
+    tune = "tune"
+
+
+_MODEL_PATTERNS = ("*.model2data.yml", "*.model2data.yaml", "*.model2data.json", "*.dbml")
+
+
+def _detect_topic(cwd: Path) -> tuple[GuideTopic, str]:
+    """`triage` when the current directory holds a model, else `setup`, and why.
+
+    Reads file names in `cwd` only: no subdirectories, no network.
+    """
+    models = sorted(path.name for pattern in _MODEL_PATTERNS for path in cwd.glob(pattern))
+    if models:
+        return GuideTopic.triage, f"found {', '.join(models)}"
+    return GuideTopic.setup, "no *.model2data.yml or *.dbml in this directory"
+
+
+@app.command("guide")
+def guide_command(
+    topic: Optional[GuideTopic] = typer.Argument(  # noqa: B008
+        None,
+        help=(
+            "setup: no model here yet -- install, write one, first run. "
+            "triage: what validate, generate and dbt build printed, and what to do. "
+            "tune: every option, model key and exit code. "
+            "Default: picked from the current directory (a *.model2data.yml or *.dbml "
+            "file in it means triage)."
+        ),
+    ),
+):
+    """Print instructions for an agent about to use model2data, as Markdown.
+
+    Reads nothing but file names in the current directory and writes nothing.
+    """
+    if topic is None:
+        topic, reason = _detect_topic(Path.cwd())
+        typer.echo(f"# model2data guide: {reason} -> {topic.value}\n")
+    page = resources.files("model2data").joinpath(f"guide/{topic.value}.md")
+    typer.echo(page.read_text("utf-8"), nl=False)

{model2data-1.10.4 → model2data-1.11.0}/model2data/dbt/hint_tests.py

@@ -14,19 +14,27 @@ What becomes a test, and what does not:
 | `generate.after: other`   | `model2data_not_before`                | none      |
 | `generate.null_rate`      | `model2data_max_null_share`            | yes       |
 | `generate.distinct: n`    | `model2data_max_distinct`              | none      |
+| `generate.when`           | `model2data_when`                      | none      |
 | table `grain`             | `model2data_unique_combination`        | none      |
 
+A column with `when` and `null_rate` has its null share tested among the rows
+`when` matches (`model2data_when_max_null_share`), since only those rows can
+hold a value, and its `model2data_when` test lets those rows be null.
+
 Enum columns keep the `accepted_values` test `generate_dbt_yml` always wrote.
 `true_rate`, `weights`, `skew`, `distribution` and the temporal shape hints
 (`business_hours`, `growth`, `seasonality`) describe a statistical shape, not a
 constraint a row can break, so they produce no test.
 
 The generic tests live in `macros/model2data_hint_tests.sql` of the generated
-project (see `write_hint_macros`): plain SQL, no dbt package.
+project (see `write_hint_macros`), and the `when` ones in
+`macros/model2data_when_tests.sql`, written only for a model that has one, so
+a project without `when` is what it was: plain SQL, no dbt package.
 """
 
 from __future__ import annotations
 
+import copy
 from collections.abc import Mapping
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -43,6 +51,8 @@ DEFAULT_TOLERANCE = 0.1
 
 MACROS_FILE = "model2data_hint_tests.sql"
 _MACROS_SOURCE = Path(__file__).parent / "templates" / "hint_macros" / MACROS_FILE
+WHEN_MACROS_FILE = "model2data_when_tests.sql"
+_WHEN_MACROS_SOURCE = Path(__file__).parent / "templates" / "hint_macros" / WHEN_MACROS_FILE
 
 
 @dataclass(frozen=True)
@@ -53,7 +63,7 @@ class HintTest:
     `column` is None for a table-level test (grain). `test` is the generic
     test's name, `arguments` its parameters as dbt's `arguments:` block, and
     `hint` the hint it comes from (`min`, `max`, `after`, `null_rate`,
-    `distinct`, `grain`). `min` and `max` of one column share one test.
+    `distinct`, `when`, `grain`). `min` and `max` of one column share one test.
     """
 
     table: str
@@ -148,14 +158,39 @@ def _column_tests(
         tests.append(HintTest(table, column.name, "model2data_not_before", arguments, "after"))
 
     null_rate = note.get("null_rate")
-    if isinstance(null_rate, (int, float)) and not isinstance(null_rate, bool):
+    if not isinstance(null_rate, (int, float)) or isinstance(null_rate, bool):
+        null_rate = None
+    conditions = _when_conditions(note.get("when"), kinds_of)
+    if conditions is not None:
+        when_arguments: dict[str, Any] = {"conditions": conditions}
+        if null_rate is not None:
+            when_arguments["required"] = False
+        tests.append(HintTest(table, column.name, "model2data_when", when_arguments, "when"))
+
+    if null_rate is not None:
         # Rounded so 0.1 + 0.1 reaches the YAML as 0.2, not 0.2000000000000001.
         limit = round(min(1.0, null_rate + tolerance), 10)
-        tests.append(
-            HintTest(
-                table, column.name, "model2data_max_null_share", {"max_share": limit}, "null_rate"
+        if conditions is None:
+            tests.append(
+                HintTest(
+                    table,
+                    column.name,
+                    "model2data_max_null_share",
+                    {"max_share": limit},
+                    "null_rate",
+                )
+            )
+        else:
+            tests.append(
+                HintTest(
+                    table,
+                    column.name,
+                    "model2data_when_max_null_share",
+                    # Its own copy, or the YAML would share the mapping as an anchor.
+                    {"conditions": copy.deepcopy(conditions), "max_share": limit},
+                    "null_rate",
+                )
             )
-        )
 
     distinct = note.get("distinct")
     if isinstance(distinct, int) and not isinstance(distinct, bool) and distinct > 0:
@@ -167,14 +202,28 @@ def _column_tests(
     return tests
 
 
-def write_hint_macros(dest: Path) -> Path:
+def _when_conditions(when: Any, kinds_of: dict[str, ColumnDef]) -> Optional[dict[str, list]]:
+    """A `when` hint as the test's `conditions`: an enum's values as the member text it holds."""
+    if not isinstance(when, dict) or not when or not all(name in kinds_of for name in when):
+        return None
+    return {
+        name: [str(value) for value in values] if kinds_of[name].enum_values else list(values)
+        for name, values in when.items()
+    }
+
+
+def write_hint_macros(dest: Path, *, when: bool = False) -> Path:
     """Write the generic tests into `dest/macros/`, and return the file's path.
 
     A project whose YAML holds `hint_tests_for` tests needs this file next to
-    it; `generate_dbt_yml` writes it itself when it emits any.
+    it; `generate_dbt_yml` writes it itself when it emits any. With `when`, the
+    file of the `when` tests is written beside it (`WHEN_MACROS_FILE`), which a
+    project holding a `model2data_when` test needs too.
     """
     macros = dest / "macros"
     macros.mkdir(parents=True, exist_ok=True)
     target = macros / MACROS_FILE
     target.write_text(_MACROS_SOURCE.read_text())
+    if when:
+        (macros / WHEN_MACROS_FILE).write_text(_WHEN_MACROS_SOURCE.read_text())
     return target

model2data-1.11.0/model2data/dbt/templates/hint_macros/model2data_when_tests.sql

@@ -0,0 +1,56 @@
+{#-
+  Generic tests model2data writes for a column's `when` hint: the column holds a
+  value only on the rows whose named columns hold one of the listed values.
+  Written beside model2data_hint_tests.sql, and only by a model that has a
+  `when`. Plain ANSI SQL, runs on DuckDB and Postgres.
+  `conditions` maps a raw column name to the values it must hold; a row whose
+  named column is null does not match.
+-#}
+
+{% macro model2data_when_literal(value) -%}
+  {%- if value is sameas true -%}true
+  {%- elif value is sameas false -%}false
+  {%- elif value is number -%}{{ value }}
+  {%- else -%}'{{ value | string | replace("'", "''") }}'
+  {%- endif -%}
+{%- endmacro %}
+
+
+{% macro model2data_when_matches(conditions) -%}
+  (case when true
+  {%- for column, values in conditions.items() %}
+    and {{ adapter.quote(column) }} in (
+      {%- for value in values -%}
+        {{ model2data_when_literal(value) }}{% if not loop.last %}, {% endif %}
+      {%- endfor -%}
+    )
+  {%- endfor %}
+  then 1 else 0 end = 1)
+{%- endmacro %}
+
+
+{#-
+  A row that does not match holds a value, or (unless `required` is false, for
+  a column with a `null_rate`) a row that matches holds none.
+-#}
+{% test model2data_when(model, column_name, conditions, required=true) %}
+select {{ column_name }} as value
+from {{ model }}
+where (not {{ model2data_when_matches(conditions) }} and {{ column_name }} is not null)
+{%- if required %}
+   or ({{ model2data_when_matches(conditions) }} and {{ column_name }} is null)
+{%- endif %}
+{% endtest %}
+
+
+{#- `null_rate` with `when`: the null share among the matching rows only. -#}
+{% test model2data_when_max_null_share(model, column_name, conditions, max_share) %}
+select null_share
+from (
+  select
+    avg(case when {{ column_name }} is null then 1.0 else 0.0 end) as null_share
+  from {{ model }}
+  where {{ model2data_when_matches(conditions) }}
+) as shares
+where null_share > {{ max_share }}
+{% endtest %}

{model2data-1.10.4 → model2data-1.11.0}/model2data/dbt/tests.py

@@ -68,7 +68,8 @@ def generate_dbt_yml(
 
     hinted = _hinted(tables, hint_tests, test_tolerance)
     if hinted:
-        write_hint_macros(dest)
+        when = any(test.hint == "when" for tests in hinted.values() for test in tests)
+        write_hint_macros(dest, when=when)
     if any(_history_entries(table) for table in tables.values()):
         write_history_macros(dest)
     fk_map = _fk_map(tables, refs)

{model2data-1.10.4 → model2data-1.11.0}/model2data/defects/checks.py

@@ -14,6 +14,10 @@ SQL reads:
   and `model2data_max_distinct`: as their macros in
   `dbt/templates/hint_macros/model2data_hint_tests.sql`, nulls skipped where the
   SQL skips them.
+- `model2data_when` and `model2data_when_max_null_share`: as their macros in
+  `dbt/templates/hint_macros/model2data_when_tests.sql`, a row matching when
+  each named column's text is one of its values (a boolean as `true`/`false`
+  in any case, a number as a number).
 - `model2data_unique_combination` and a composite key's `unique_combination`: a
   combination of values on two rows, nulls grouping together as in `group by`.
 - A history table's `model2data_one_current_row` (a key with other than one
@@ -342,6 +346,49 @@ def _max_null_share(test: DbtTest, frame: pd.DataFrame, seeded: Mapping, cache:
     return nulls / len(frame) > test.arguments["max_share"]
 
 
+def _matches(test: DbtTest, frame: pd.DataFrame, cache: Columns) -> list[bool]:
+    """Per row: does each column of the test's `conditions` hold one of its values?"""
+    result = [True] * len(frame)
+    for column, values in test.arguments["conditions"].items():
+        texts = {value for value in values if isinstance(value, str)}
+        flags = {str(value).lower() for value in values if isinstance(value, bool)}
+        numbers = {
+            float(value)
+            for value in values
+            if isinstance(value, (int, float)) and not isinstance(value, bool)
+        }
+        for row, text in enumerate(cache.values(frame, column)):
+            if text is None or not (
+                text in texts or text.lower() in flags or _number(text) in numbers
+            ):
+                result[row] = False
+    return result
+
+
+def _when(test: DbtTest, frame: pd.DataFrame, seeded: Mapping, cache: Columns) -> bool:
+    required = test.arguments.get("required", True)
+    values = cache.values(frame, str(test.column))
+    for matches, value in zip(_matches(test, frame, cache), values, strict=True):
+        if (not matches and value is not None) or (required and matches and value is None):
+            return True
+    return False
+
+
+def _when_max_null_share(
+    test: DbtTest, frame: pd.DataFrame, seeded: Mapping, cache: Columns
+) -> bool:
+    values = cache.values(frame, str(test.column))
+    held = [
+        value
+        for matches, value in zip(_matches(test, frame, cache), values, strict=True)
+        if matches
+    ]
+    if not held:
+        return False
+    nulls = sum(value is None for value in held)
+    return nulls / len(held) > test.arguments["max_share"]
+
+
 def _max_distinct(test: DbtTest, frame: pd.DataFrame, seeded: Mapping, cache: Columns) -> bool:
     return len(cache.number_set(frame, str(test.column))) > test.arguments["max_count"]
 
@@ -388,6 +435,8 @@ _CHECKS: dict[str, Callable[[DbtTest, pd.DataFrame, Mapping, Columns], bool]] =
     "model2data_not_before": _not_before,
     "model2data_max_null_share": _max_null_share,
     "model2data_max_distinct": _max_distinct,
+    "model2data_when": _when,
+    "model2data_when_max_null_share": _when_max_null_share,
     "model2data_unique_combination": _unique_combination,
     "unique_combination": _unique_combination,
     "model2data_one_current_row": _one_current_row,