model2data 1.16.2__tar.gz → 1.16.4__tar.gz

model2data-1.16.4/PKG-INFO

@@ -0,0 +1,406 @@
+Metadata-Version: 2.4
+Name: model2data
+Version: 1.16.4
+Summary: Turn a data model into realistic synthetic data, a runnable dbt project and metrics with known values
+Author: JB Analytica
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/JB-Analytica/model2data
+Project-URL: Studio, https://studio.jbanalytica.com
+Project-URL: Repository, https://github.com/JB-Analytica/model2data
+Project-URL: Issues, https://github.com/JB-Analytica/model2data/issues
+Project-URL: Changelog, https://github.com/JB-Analytica/model2data/blob/main/CHANGELOG.md
+Keywords: dbt,dbml,synthetic-data,test-data,analytics-engineering,data-engineering,duckdb,seed-data
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: dbt-core>=1.11
+Requires-Dist: dbt-duckdb>=1.11
+Requires-Dist: faker>=37.12.0
+Requires-Dist: jsonschema>=4.18
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: pyparsing<4,>=3.1
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: typer>=0.20.0
+Provides-Extra: postgres
+Requires-Dist: dbt-postgres>=1.11; extra == "postgres"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pre-commit; python_version >= "3.11" and extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: ty>=0.0.4; extra == "dev"
+Requires-Dist: types-pyyaml; extra == "dev"
+Requires-Dist: poethepoet>=0.38.0; extra == "dev"
+Dynamic: license-file
+
+# model2data
+
+[![PyPI](https://img.shields.io/pypi/v/model2data)](https://pypi.org/project/model2data/)
+[![PyPI downloads](https://img.shields.io/pypi/dm/model2data)](https://pypi.org/project/model2data/)
+[![Python versions](https://img.shields.io/pypi/pyversions/model2data)](https://pypi.org/project/model2data/)
+[![CI](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml/badge.svg)](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/JB-Analytica/model2data/branch/main/graph/badge.svg)](https://codecov.io/gh/JB-Analytica/model2data)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/JB-Analytica/model2data/blob/main/LICENSE)
+
+**Realistic, test-passing data for any data model, and the dbt project around it, in one command.**
+
+You describe the tables. model2data generates data whose foreign keys resolve, whose dates make
+sense and whose revenue adds up to a number it tells you in advance, and it writes a dbt project
+that loads, builds and tests that data with one `dbt build`. No production access, no
+hand-made CSVs, no dbt boilerplate.
+
+```bash
+pip install model2data
+model2data --file examples/ecommerce.model2data.yml --rows 200 --seed 42   # examples/ is in the GitHub repository
+cd dbt_ecommerce && dbt build                                               # PASS=59 ERROR=0
+```
+
+![model2data generating a project with metrics and running it with dbt](https://raw.githubusercontent.com/JB-Analytica/model2data/main/assets/demo.gif)
+
+> **Rather design the model than type it?** [**model2data studio**](https://studio.jbanalytica.com)
+> is this engine in the browser: draw the schema, see every column's values before you generate,
+> shape the data on a sample of your own, and export a dbt project with a star schema or Data
+> Vault on top. [Start free, no install →](https://studio.jbanalytica.com)
+
+---
+
+## The problem
+
+Every data project starts the same way: the model is ready and the data isn't.
+
+- **Production data is off-limits.** Privacy, a pending contract, a source system nobody has
+  access to yet. The pipeline waits; the client demo waits.
+- **Hand-made mock data breaks on contact.** Two CSVs are fine. Ten tables later, orders point at
+  customers who don't exist, shipments leave before they are ordered, and every `relationships`
+  test fails for reasons that have nothing to do with your SQL.
+- **Random data proves nothing.** A dashboard over `Lorem ipsum` and uniform noise can't tell you
+  whether revenue is right, because nobody knows what revenue *should* be.
+- **Fixtures rot.** A CSV committed once is right the day it was exported; after that it drifts
+  from the model, and CI tests against the past.
+
+So teams spend days on throwaway data, or ship pipelines tested against nothing.
+
+## The solution
+
+Write the model once. model2data reads it and produces data that behaves like the real thing, and
+the dbt project to prove it:
+
+| You need | model2data gives you |
+| --- | --- |
+| Data that looks real | `email` gets emails, `city` gets cities, `sku ean13` gets barcodes: about 40 name patterns and every Faker provider, in any locale. |
+| Joins that work | Tables generated in dependency order; every foreign key hits a row that exists. An order never predates its customer (`after: customers.created_at`). |
+| A project, not a pile of files | Seeds, staging models, `not_null`/`unique`/`relationships`/`accepted_values` tests and a DuckDB or Postgres profile. `dbt build` goes green. |
+| Numbers you can check | A metrics file defines revenue once; the run writes its exact value per month and a dbt test that the warehouse must reproduce. |
+| Fixtures you can commit | Same model, seed and `--as-of`: the same bytes, on any machine, any day. |
+| Data that fails on purpose | `--defects training` breaks each standard dbt test once and writes down which. |
+
+Nothing but a schema goes in; nothing but synthetic data comes out. It is MIT licensed and runs
+anywhere Python 3.10 to 3.14 does.
+
+---
+
+## Before and after
+
+In: a model (here the `orders` table of
+[`examples/ecommerce.model2data.yml`](https://github.com/JB-Analytica/model2data/blob/main/examples/ecommerce.model2data.yml)) and, optionally, the
+metrics that matter ([`examples/ecommerce.metrics.yml`](https://github.com/JB-Analytica/model2data/blob/main/examples/ecommerce.metrics.yml)). DBML,
+hand-written or exported from an existing database, works as input too.
+
+```yaml
+  orders:
+    columns:
+      id: {type: bigint, pk: true, not_null: true}
+      customer_id: {type: bigint, not_null: true, references: customers.id}
+      order_date: {type: timestamp, not_null: true}
+      total_amount:
+        type: numeric
+        not_null: true
+        generate: {min: 10, max: 5000}
+      status: order_status
+```
+
+```yaml
+metrics:
+  revenue:
+    measure: orders.total_amount
+    agg: sum
+    where:
+      orders.status: [processing, shipped, delivered]
+    time: orders.order_date
+```
+
+```bash
+model2data --file examples/ecommerce.model2data.yml --metrics examples/ecommerce.metrics.yml \
+  --rows 200 --seed 42 --as-of 2026-01-31
+```
+
+Out: a dbt project that runs.
+
+```
+dbt_ecommerce/
+├── seeds/raw/                  customers.csv, orders.csv, order_items.csv, products.csv, ...
+├── models/staging/             stg_<table>.sql + stg_<table>.yml (with tests), for each table
+├── data-tests/metrics/         one test per metric: the warehouse must match the known value
+├── osi/ecommerce.yml           the model and its metrics as Apache Ossie
+├── metric_values.json          each metric's known value, overall and per month
+├── dbt_project.yml
+└── profiles.yml                DuckDB, zero config
+```
+
+```
+Finished running 5 seeds, 49 data tests, 5 view models in 0 hours 0 minutes and 0.71 seconds (0.71s).
+
+Done. PASS=59 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=59
+```
+
+And `metric_values.json` says what a dashboard on this data must show: revenue 220,291.80 over
+92 orders, month by month. If your semantic layer or BI tool says otherwise, the bug is in the
+SQL, and you know it before a stakeholder does.
+
+---
+
+## Who it's for
+
+- **Consultants who need a demo before source access.** Write the client's model after the
+  kick-off call, generate a year of their business, and show a working dbt project and dashboard
+  in week one, without touching production.
+- **Analytics engineers who want real fixtures in CI.** Commit the model, generate with `--seed`
+  and `--as-of`, and every CI run builds against the same bytes. `--days` hands an incremental
+  model tomorrow's batch.
+- **Trainers who need data that fails for the right reason.** `--defects training` makes
+  `unique`, `not_null`, `relationships` and `accepted_values` each fail once, and says where.
+- **Teams who define metrics once.** Write revenue in one file, let `dbt build` prove the SQL,
+  and hand the definitions to Apache Ossie or Lightdash.
+
+---
+
+## model2data studio: design the model, ship the project
+
+The CLI is the engine. [**model2data studio**](https://studio.jbanalytica.com) is where models get
+designed, shown to a client and handed to a team, and it is built by the same people on the same
+engine, so what you preview is byte for byte what you export.
+
+![model2data studio: the model editor beside its live diagram](https://raw.githubusercontent.com/JB-Analytica/model2data/main/assets/studio-editor-and-diagram.png)
+
+What it adds to the CLI:
+
+- **A live diagram as you type.** Write YAML or DBML (dbdiagram schemas work unchanged), or paste
+  SQL `CREATE TABLE` statements, and the entity diagram redraws as you type. Click a column to
+  see everything that joins to it. Drag a column onto another to draw the reference.
+- **See the data before you generate it.** Every column shows an example of the value it will
+  get, and columns nothing recognises are flagged, so placeholder data never slips through.
+- **Shape it like your data.** Drop in a CSV sample, or just summary statistics when even a sample
+  can't leave the warehouse, and the studio proposes ranges, distributions, null rates, enum
+  weights, seasonality and skew, each with a before-and-after chart. The file is profiled in your
+  browser and never uploaded.
+- **More than staging.** Export the dbt project with a Kimball star schema, a Data Vault, or a
+  dbt semantic layer on top, real models with tests, derived from the same schema.
+- **Regenerate in CI.** An API key, the `model2data-studio` CLI and an MCP server let a pipeline
+  or a coding agent pull fresh data from the saved project, so fixtures follow the model instead
+  of the last export.
+- **Share it.** A read-only link that also embeds as a clean diagram in Notion, Confluence or a
+  wiki.
+
+| | CLI (this repo) | Studio Free | Studio Pro |
+| --- | --- | --- | --- |
+| Engine: realistic, FK-correct, deterministic data | ✓ | ✓ | ✓ |
+| Live diagram, value previews, model checks | | ✓ | ✓ |
+| Rows per table | unlimited | 100 | 50,000 |
+| Saved projects | | 2 | unlimited |
+| Share links and embeds | | ✓ | ✓ |
+| CSV export | ✓ | ✓ | ✓ |
+| dbt project download | ✓ (staging) | preview | ✓ |
+| Shape from a sample, visual shaping | | preview | ✓ |
+| Star schema, Data Vault, semantic layer | | preview | ✓ |
+| API keys, CI pulls, MCP server | | | ✓ |
+
+**Pro is EUR 99 a month** (or EUR 990 a year); **Team is EUR 399 a month for five seats.**
+For a consultant, that's less than an hour of the time it replaces on the first project.
+
+**[Open the studio, free →](https://studio.jbanalytica.com)** ·
+[Plans and pricing](https://www.jbanalytica.com/model2data/pricing/)
+
+Keep using the CLI for scripting and committed fixtures; it stays free and open source. Use the
+studio when the model is still being designed, when someone else needs to see it, or when the
+project needs more than a staging layer.
+
+---
+
+## Everything the engine does
+
+**Values that look real.** About 40 column-name patterns (`email`, `first_name`, `city`, `phone`,
+`company`, ...), any Faker provider as a type (`billing_country state`, `sku ean13`), and
+`--locale nl_BE` for people and addresses from one country. → [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md)
+
+**Foreign keys that resolve.** Every foreign key points at a parent row that exists, onto a
+primary or a unique key; composite keys hold; `after: customers.created_at` keeps a child's date
+after its parent's; a column that repeats a parent's value through a foreign key holds the right
+parent's value. → [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md)
+
+**A dbt project that runs.** Seeds, staging models, schema tests, tests from the model's own hints,
+optional unit test fixtures (`--unit-tests`), and a DuckDB or Postgres profile.
+→ [The generated dbt project](https://github.com/JB-Analytica/model2data/blob/main/docs/dbt-project.md)
+
+**The same files, byte for byte.** Same model, `--seed`, `--as-of` and options, same files, with
+the same model2data version. `--table-seed orders=7` re-rolls one table and leaves the rest alone.
+→ [The determinism promise](https://github.com/JB-Analytica/model2data/blob/main/docs/generating.md#the-determinism-promise)
+
+**Days after the first.** `--days 7` or `--next` adds new rows and updates existing ones; enum
+`transitions` move an order from `pending` to `shipped`; each day is a batch or a changelog.
+→ [Days after the first](https://github.com/JB-Analytica/model2data/blob/main/docs/days.md)
+
+**History tables.** `history: true` writes every version of every row with `valid_from`,
+`valid_to` and `is_current`, an SCD type 2 source. → [History tables](https://github.com/JB-Analytica/model2data/blob/main/docs/history.md)
+
+**When things happen.** `--business-hours`, `--growth` and `--seasonality` shape timestamps;
+`updated_at` never precedes `created_at`; `when` fills `shipped_at` only on shipped orders.
+→ [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md)
+
+**How the data is spread.** `--skew` lets a few customers place most orders; `weights`,
+`true_rate`, `null_rate`, `distinct` and normal, lognormal or exponential `distribution`s shape a
+column. → [How the data is spread](https://github.com/JB-Analytica/model2data/blob/main/docs/distributions.md)
+
+**Defects on purpose.** `--defects training` or `messy`, or a table's own list: duplicate keys,
+orphans, nulls, invalid values, messy text, late-arriving rows. `EXPECTED_FAILURES.md` says what
+should fail. → [Defects on purpose](https://github.com/JB-Analytica/model2data/blob/main/docs/defects.md)
+
+**Metrics with known values.** Each metric's value in `metric_values.json`, a dbt test per metric,
+an Apache Ossie semantic model, and with `--lightdash` the same metrics as Lightdash reads them.
+→ [Metrics with known values](https://github.com/JB-Analytica/model2data/blob/main/docs/metrics.md)
+
+**Validate in CI.** `model2data validate` prints GitHub annotations; `uses: JB-Analytica/model2data@v1`
+fails a pull request that breaks a model. Pre-commit and GitLab CI too. → [Validate models in CI](https://github.com/JB-Analytica/model2data/blob/main/docs/ci.md)
+
+**Use it from an agent.** `uvx model2data@latest guide` prints instructions written for a coding
+agent; [LLMS.md](https://github.com/JB-Analytica/model2data/blob/main/LLMS.md) takes an LLM from a plain-English description to a running project.
+→ [Agents and LLMs](https://github.com/JB-Analytica/model2data/blob/main/docs/agents.md)
+
+---
+
+## How it works
+
+```
+model (.model2data.yml or DBML) + metrics file (optional)
+  -> read and validate -> generate -> scaffold the dbt project + compute each metric's known value
+  -> seeds, staging models and tests, metric tests, metric_values.json, osi/, profiles.yml
+  -> dbt build
+```
+
+1. **Read.** The model is checked against [the spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md) (DBML is converted
+   to the same model first), and a metrics file against [its spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/metrics/README.md)
+   and the model. A model with errors is refused with every issue and where it is.
+2. **Generate.** Typed generation for SQL types, then a Faker provider named as the type, then
+   name-aware inference; foreign keys resolve against parent rows already generated. The metrics
+   file is never read here, so the data is the same with or without it.
+3. **Scaffold.** CSV seeds, staging models that `ref()` them, schema and hint tests, descriptions
+   from the model, and a DuckDB or Postgres profile.
+4. **Compute metrics.** Each metric's known value, a dbt test that checks it, the Apache Ossie
+   file and, with `--lightdash`, Lightdash metrics in the staging YAML.
+
+---
+
+## Installation
+
+```bash
+pip install model2data                 # DuckDB included
+pip install "model2data[postgres]"     # to target Postgres with --adapter postgres
+uvx model2data --help                  # or run it without installing
+```
+
+Python 3.10 to 3.14. The examples used on this page are in [`examples/`](https://github.com/JB-Analytica/model2data/tree/main/examples/).
+
+Or skip the install: [model2data studio](https://studio.jbanalytica.com) runs the same engine in
+the browser.
+
+---
+
+## Documentation
+
+- [The model file](https://github.com/JB-Analytica/model2data/blob/main/docs/model-file.md): `.model2data.yml`, DBML as input, `validate`, `convert`
+- [Generating a project](https://github.com/JB-Analytica/model2data/blob/main/docs/generating.md): row counts, `--as-of`, the determinism promise, `--table-seed`, `--locale`
+- [The generated dbt project](https://github.com/JB-Analytica/model2data/blob/main/docs/dbt-project.md): `dbt build`, Postgres, unit tests, hint tests, structure
+- [Days after the first](https://github.com/JB-Analytica/model2data/blob/main/docs/days.md) and [history tables](https://github.com/JB-Analytica/model2data/blob/main/docs/history.md)
+- [When things happen](https://github.com/JB-Analytica/model2data/blob/main/docs/time-shapes.md) and [how the data is spread](https://github.com/JB-Analytica/model2data/blob/main/docs/distributions.md)
+- [Defects on purpose](https://github.com/JB-Analytica/model2data/blob/main/docs/defects.md)
+- [Metrics with known values](https://github.com/JB-Analytica/model2data/blob/main/docs/metrics.md)
+- [Validate models in CI](https://github.com/JB-Analytica/model2data/blob/main/docs/ci.md)
+- [Agents and LLMs](https://github.com/JB-Analytica/model2data/blob/main/docs/agents.md), and [LLMS.md](https://github.com/JB-Analytica/model2data/blob/main/LLMS.md)
+- The [model spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/README.md) and the [metrics spec](https://github.com/JB-Analytica/model2data/blob/main/model2data/spec/metrics/README.md)
+- [Examples](https://github.com/JB-Analytica/model2data/blob/main/examples/README.md) and the [changelog](https://github.com/JB-Analytica/model2data/blob/main/CHANGELOG.md)
+
+---
+
+## dbt-core versions
+
+model2data requires **dbt-core >= 1.11**, tracking [dbt's own version support
+policy](https://docs.getdbt.com/docs/dbt-versions): 1.11 is the oldest minor release dbt Labs
+still supports. Generated projects build without deprecation warnings on every supported
+version, and CI proves it on each push with a real `dbt build` against both the floor and the
+newest release. Pinned to an older dbt-core? model2data 0.5.x supports down to 1.8.5.
+
+---
+
+## Design decisions and limitations
+
+- **DuckDB by default**, because it needs no setup. Postgres is supported with
+  `--adapter postgres`; other adapters need their profile adjusted by hand.
+- **Deterministic synthetic data**, for testing, demos and prototyping. It is not a replacement
+  for real data, a migration tool or a real-time generator.
+- **Metrics describe, never steer.** Adding or changing a metric changes no generated data, only
+  the known values and tests computed from it.
+- **Generation is heuristic.** It looks real and keeps every key and rule the model states, but it
+  won't reproduce every quirk of a production distribution. (The studio's *Shape from a sample*
+  gets closer, from your own data's statistics.)
+- **The generated project stops at staging.** Marts are yours to write, or the studio's Pro export
+  adds a star schema, Data Vault or semantic layer.
+- **Composite foreign keys** are generated as independent single-column keys: each column is valid,
+  the combination only when the child declares it as a key (`keys`).
+- A reference onto a column that is not a key is allowed with a warning; its values are drawn
+  from the ones the parent column holds.
+
+---
+
+## Project status
+
+model2data is stable and actively developed, and follows semantic versioning. CI runs a real
+`dbt build` against the oldest supported and the newest dbt-core on every push and checks the
+output of a spread of models byte for byte. The [changelog](https://github.com/JB-Analytica/model2data/blob/main/CHANGELOG.md) names every release
+that changes what a seed produces.
+
+Open for contributions: more database adapters (Snowflake, BigQuery), and example mart-layer
+models on top of staging.
+
+## Contributing
+
+Issues and pull requests are welcome: bugs, feature requests, example models, generators. See
+[CONTRIBUTING.md](https://github.com/JB-Analytica/model2data/blob/main/CONTRIBUTING.md) for guidelines, [DEVELOPMENT.md](https://github.com/JB-Analytica/model2data/blob/main/DEVELOPMENT.md) for the local
+setup and release process, and our [Code of Conduct](https://github.com/JB-Analytica/model2data/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+MIT. See [LICENSE](https://github.com/JB-Analytica/model2data/blob/main/LICENSE).
+
+---
+
+<p align="center">
+  <a href="https://www.jbanalytica.com">
+    <img src="https://raw.githubusercontent.com/JB-Analytica/model2data/main/assets/jba-icon-dark-bg.svg" alt="JB Analytica" height="40">
+  </a>
+  <br>
+  Built and maintained by <a href="https://www.jbanalytica.com"><strong>JB Analytica</strong></a>:
+  Data & Analytics Engineering · Data Platform Architecture · Modern BI.
+  <br>
+  Designing a model? Try <a href="https://studio.jbanalytica.com"><strong>model2data studio</strong></a>, free in the browser.
+</p>