model2data 1.13.0__tar.gz → 1.14.0__tar.gz

{model2data-1.13.0/model2data.egg-info → model2data-1.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: model2data
-Version: 1.13.0
+Version: 1.14.0
 Summary: Turn a data model into realistic synthetic data, a runnable dbt project and metrics with known values
 Author: JB Analytica
 License-Expression: MIT
@@ -131,6 +131,9 @@ orphans, nulls, invalid values, messy text, late-arriving rows. Every run with d
 orders, average order value. The run writes each metric's value over the generated data to
 `metric_values.json`, overall and per month, a dbt test per metric that must reproduce it on the
 warehouse, and an [Apache Ossie](https://github.com/apache/ossie) semantic model under `osi/`.
+New in 1.14.0: `--lightdash` writes the same metrics into the staging models' YAML as
+[Lightdash](https://www.lightdash.com/) reads them, filters, joins and ratios included, so the
+dashboard's revenue can be checked against the number the run already knows.
 → [Metrics with known values](https://github.com/JB-Analytica/model2data/blob/main/docs/metrics.md)
 
 **Validate in CI.** `model2data validate` checks models and metrics files and prints GitHub
@@ -272,7 +275,7 @@ definition goes in; nothing but synthetic data comes out.
   `accepted_values` fail for the right reason.
 - **A team defining metrics once.** Write revenue in one metrics file, check every number the
   semantic layer or dashboard shows against `metric_values.json`, let `dbt build` prove the SQL,
-  and hand the definitions on as Apache Ossie.
+  and hand the definitions on as Apache Ossie or straight to Lightdash.
 
 ## How it works
 
@@ -298,7 +301,8 @@ model (.model2data.yml or DBML) + metrics file (optional)
    model's hints, table and column `description:` fields from the model, and a profile for
    DuckDB (zero-config, file-based) or Postgres.
 4. **Compute metrics.** With `--metrics`, computes each metric over the generated data and writes
-   its known value, a dbt test that checks it, and the Apache Ossie file.
+   its known value, a dbt test that checks it, and the Apache Ossie file; with `--lightdash`,
+   Lightdash metrics and dimensions in the staging models' YAML.
 
 ---
 

{model2data-1.13.0 → model2data-1.14.0}/README.md

@@ -84,6 +84,9 @@ orphans, nulls, invalid values, messy text, late-arriving rows. Every run with d
 orders, average order value. The run writes each metric's value over the generated data to
 `metric_values.json`, overall and per month, a dbt test per metric that must reproduce it on the
 warehouse, and an [Apache Ossie](https://github.com/apache/ossie) semantic model under `osi/`.
+New in 1.14.0: `--lightdash` writes the same metrics into the staging models' YAML as
+[Lightdash](https://www.lightdash.com/) reads them, filters, joins and ratios included, so the
+dashboard's revenue can be checked against the number the run already knows.
 → [Metrics with known values](docs/metrics.md)
 
 **Validate in CI.** `model2data validate` checks models and metrics files and prints GitHub
@@ -225,7 +228,7 @@ definition goes in; nothing but synthetic data comes out.
   `accepted_values` fail for the right reason.
 - **A team defining metrics once.** Write revenue in one metrics file, check every number the
   semantic layer or dashboard shows against `metric_values.json`, let `dbt build` prove the SQL,
-  and hand the definitions on as Apache Ossie.
+  and hand the definitions on as Apache Ossie or straight to Lightdash.
 
 ## How it works
 
@@ -281,7 +284,8 @@ flowchart LR
    model's hints, table and column `description:` fields from the model, and a profile for
    DuckDB (zero-config, file-based) or Postgres.
 4. **Compute metrics.** With `--metrics`, computes each metric over the generated data and writes
-   its known value, a dbt test that checks it, and the Apache Ossie file.
+   its known value, a dbt test that checks it, and the Apache Ossie file; with `--lightdash`,
+   Lightdash metrics and dimensions in the staging models' YAML.
 
 ---
 

{model2data-1.13.0 → model2data-1.14.0}/README_PYPI.md

@@ -84,6 +84,9 @@ orphans, nulls, invalid values, messy text, late-arriving rows. Every run with d
 orders, average order value. The run writes each metric's value over the generated data to
 `metric_values.json`, overall and per month, a dbt test per metric that must reproduce it on the
 warehouse, and an [Apache Ossie](https://github.com/apache/ossie) semantic model under `osi/`.
+New in 1.14.0: `--lightdash` writes the same metrics into the staging models' YAML as
+[Lightdash](https://www.lightdash.com/) reads them, filters, joins and ratios included, so the
+dashboard's revenue can be checked against the number the run already knows.
 → [Metrics with known values](https://github.com/JB-Analytica/model2data/blob/main/docs/metrics.md)
 
 **Validate in CI.** `model2data validate` checks models and metrics files and prints GitHub
@@ -225,7 +228,7 @@ definition goes in; nothing but synthetic data comes out.
   `accepted_values` fail for the right reason.
 - **A team defining metrics once.** Write revenue in one metrics file, check every number the
   semantic layer or dashboard shows against `metric_values.json`, let `dbt build` prove the SQL,
-  and hand the definitions on as Apache Ossie.
+  and hand the definitions on as Apache Ossie or straight to Lightdash.
 
 ## How it works
 
@@ -251,7 +254,8 @@ model (.model2data.yml or DBML) + metrics file (optional)
    model's hints, table and column `description:` fields from the model, and a profile for
    DuckDB (zero-config, file-based) or Postgres.
 4. **Compute metrics.** With `--metrics`, computes each metric over the generated data and writes
-   its known value, a dbt test that checks it, and the Apache Ossie file.
+   its known value, a dbt test that checks it, and the Apache Ossie file; with `--lightdash`,
+   Lightdash metrics and dimensions in the staging models' YAML.
 
 ---
 

{model2data-1.13.0 → model2data-1.14.0}/model2data/cli.py

@@ -50,13 +50,16 @@ from model2data.metrics import (
     SPEC_VERSION as METRICS_SPEC_VERSION,
 )
 from model2data.metrics import (
+    LightdashExport,
     Metrics,
     MetricsError,
     is_metrics_file,
     known_values,
     metrics_stem,
     sibling_model,
+    to_lightdash,
     to_ossie,
+    write_lightdash,
     write_metric_tests,
 )
 from model2data.metrics import load as load_metrics
@@ -471,6 +474,16 @@ def main(
             "without it. Not picked up from beside the model: pass it."
         ),
     ),
+    lightdash: bool = typer.Option(
+        False,
+        "--lightdash",
+        help=(
+            "Add Lightdash metrics and dimensions to the staging models' YAML\n"
+            "(models/staging/stg_*.yml, under config.meta): the --metrics file's metrics,\n"
+            "or without one the metrics the model implies. Without it those files are\n"
+            "unchanged."
+        ),
+    ),
 ):
     """
     Generate synthetic data and a dbt project from a model.
@@ -726,6 +739,13 @@ def main(
     # Metrics: only added files, computed from what the seeds hold
     # -------------------------
     metrics_count: Optional[int] = None
+    lightdash_export = None
+    if _given(lightdash):
+        typer.echo("🔦 Adding Lightdash metrics and dimensions to the staging models...")
+        lightdash_export = to_lightdash(
+            resolve_metrics(model, metrics_doc, model_name=_model_stem(file))
+        )
+        write_lightdash(dest, lightdash_export)
     if metrics_doc is not None and metrics_file is not None:
         typer.echo("📏 Computing each metric's known value and its dbt test...")
         semantic = resolve_metrics(model, metrics_doc, model_name=_model_stem(file))
@@ -797,6 +817,13 @@ def main(
             f"osi/{project_name}.yml, data-tests/metrics/)"
         )
 
+    if lightdash_export is not None:
+        typer.echo(
+            f"  Lightdash:               {len(lightdash_export.metrics)} metrics "
+            "(models/staging/stg_*.yml, config.meta)"
+        )
+        _print_losses("Not expressible in Lightdash, left out", lightdash_export.lossiness)
+
     if report is not None:
         _print_defects(report)
 
@@ -1074,7 +1101,14 @@ metrics_app = typer.Typer(
 )
 app.add_typer(metrics_app, name="metrics")
 
-_METRICS_EXPORTS = ("ossie",)
+_METRICS_EXPORTS = ("ossie", "lightdash")
+
+
+def _print_losses(heading: str, losses: list) -> None:
+    if losses:
+        typer.echo(f"ℹ️  {heading}:")
+        for loss in losses:
+            typer.echo(f"  - {loss}")
 
 
 def _metrics_inputs(file: Path, metrics_file: Optional[Path]) -> Any:
@@ -1138,7 +1172,14 @@ def metrics_list_command(
 def metrics_export_command(
     file: Path = _MODEL_OPTION,
     metrics_file: Optional[Path] = _METRICS_OPTION,
-    to: str = typer.Option("ossie", "--to", help="The format: 'ossie' (Apache Ossie 0.1.1, YAML)."),
+    to: str = typer.Option(
+        "ossie",
+        "--to",
+        help=(
+            "The format: 'ossie' (Apache Ossie 0.1.1, YAML) or 'lightdash' (dbt model "
+            "properties with Lightdash's config.meta, YAML)."
+        ),
+    ),
     output: Optional[Path] = typer.Option(  # noqa: B008
         None,
         "--output",
@@ -1153,6 +1194,11 @@ def metrics_export_command(
     Ossie datasets read the staging models of the dbt project `generate` writes
     (staging.stg_<table>). What Ossie 0.1.1 cannot say is listed at the top of the file,
     and carried in custom_extensions.
+
+    Lightdash's file describes those staging models (stg_<table>) as a dbt properties
+    file. dbt reads a model's properties from one file only, so in a project `generate`
+    wrote, use `generate --lightdash` instead: it merges the same meta into the staging
+    models' own YAML. What Lightdash cannot say is listed at the top of the file.
     """
     to = (_given(to) or "ossie").lower()
     if to not in _METRICS_EXPORTS:
@@ -1160,7 +1206,7 @@ def metrics_export_command(
             f"Choose one of: {', '.join(_METRICS_EXPORTS)}.", param_hint="--to"
         )
     semantic = _metrics_inputs(file, _given(metrics_file))
-    export = to_ossie(semantic)
+    export = to_ossie(semantic) if to == "ossie" else to_lightdash(semantic)
     text = export.to_yaml()
     output = _given(output)
     if output is None:
@@ -1170,8 +1216,11 @@ def metrics_export_command(
         typer.echo(f"❌ {output} already exists. Use --force to overwrite it.")
         raise typer.Exit(1)
     output.write_text(text, encoding="utf-8")
+    if isinstance(export, LightdashExport):
+        typer.echo(f"✅ Wrote {output} (Lightdash, dbt YAML, {len(export.metrics)} metrics)")
+        _print_losses("Not expressible in Lightdash, left out", export.lossiness)
+        return
     typer.echo(f"✅ Wrote {output} (Apache Ossie 0.1.1, {len(semantic.metrics)} metrics)")
-    if export.lossiness:
-        typer.echo("ℹ️  Not expressible in Ossie 0.1.1 itself, carried in custom_extensions:")
-        for loss in export.lossiness:
-            typer.echo(f"  - {loss}")
+    _print_losses(
+        "Not expressible in Ossie 0.1.1 itself, carried in custom_extensions", export.lossiness
+    )

{model2data-1.13.0 → model2data-1.14.0}/model2data/guide/tune.md

@@ -8,7 +8,7 @@
 | `model2data validate M...` | check models against spec 0.4.0, and `*.metrics.yml` files against the metrics spec and their model (`--model M`, else the model beside it with the same stem) | 0 conforms (warnings allowed), 1 any error |
 | `model2data convert M` | print M as `.model2data.yml`; `-o FILE` writes it, `--force` overwrites | 0 ok, 1 unreadable model or output exists |
 | `model2data metrics list -f M [--metrics F]` | every metric: the file's, then the inferred ones | 0, 1 on a model or metrics error |
-| `model2data metrics export -f M [--metrics F] --to ossie` | the model and its metrics as Apache Ossie 0.1.1; `-o FILE` writes it, `--force` overwrites | 0, 1 on an error or an existing output, 2 on another `--to` |
+| `model2data metrics export -f M [--metrics F] --to ossie\|lightdash` | the model and its metrics as Apache Ossie 0.1.1, or as Lightdash meta in one dbt properties file for the staging models; `-o FILE` writes it, `--force` overwrites | 0, 1 on an error or an existing output, 2 on another `--to` |
 | `model2data guide [TOPIC]` | this page; topics `setup`, `triage`, `tune` | 0 |
 
 `generate` exits 0 on success; 1 on a model error, an existing `dbt_<name>/` without
@@ -142,7 +142,9 @@ Only tables with `incremental` change after the first day.
 With `--metrics FILE`, and nothing else changed: `metric_values.json` (each metric's known
 value over the seeds, overall and by month), `data-tests/metrics/metric_<name>.sql` (a dbt
 test per metric that fails unless the warehouse gets that value) and `osi/<name>.yml`
-(Apache Ossie 0.1.1).
+(Apache Ossie 0.1.1). With `--lightdash` (with or without `--metrics`), the staging models'
+`stg_<table>.yml` also carry Lightdash metrics, dimensions and joins under `config.meta`; what
+Lightdash cannot say is listed in the summary.
 
 ## Metrics file
 

{model2data-1.13.0 → model2data-1.14.0}/model2data/metrics/__init__.py

@@ -12,6 +12,7 @@ generator never reads it: a model generates the same data with or without one.
     >>> values = metrics.known_values(semantic, frames)     # over a run's generated tables
     >>> export = metrics.to_ossie(semantic)                 # Apache Ossie 0.1.1
     >>> export.to_yaml(), export.lossiness
+    >>> metrics.to_lightdash(semantic).to_yaml()           # Lightdash meta, dbt YAML
 
 `resolve(model)` with no metrics file gives the metrics the model implies on
 its own: one per `measure` column, and a row count per fact.
@@ -23,6 +24,7 @@ from model2data.metrics.expression import ExpressionError
 from model2data.metrics.expression import parse as parse_expression
 from model2data.metrics.graph import Graph, Join
 from model2data.metrics.infer import inferred_metrics
+from model2data.metrics.lightdash import LightdashExport, to_lightdash, write_lightdash
 from model2data.metrics.ossie import OSSIE_VERSION, Loss, OssieExport, to_ossie
 from model2data.metrics.reader import (
     MetricsError,
@@ -71,6 +73,7 @@ __all__ = [
     "Join",
     "KnownValue",
     "KnownValues",
+    "LightdashExport",
     "Loss",
     "Measure",
     "Metric",
@@ -94,7 +97,9 @@ __all__ = [
     "parse_expression",
     "resolve",
     "sibling_model",
+    "to_lightdash",
     "to_ossie",
     "validate",
+    "write_lightdash",
     "write_metric_tests",
 ]