model2data 0.1.0__tar.gz

model2data-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 JB Analytica
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

model2data-0.1.0/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: model2data
+Version: 0.1.0
+Summary: Generate analytics-ready datasets from DBML models
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: dbt-core>=1.5.0
+Requires-Dist: dbt-duckdb>=1.5.0
+Requires-Dist: faker>=37.12.0
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: pyyaml>=6.0.3
+Requires-Dist: typer>=0.20.0
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# model2data
+
+[![PyPI](https://img.shields.io/pypi/v/model2data)](https://pypi.org/project/model2data/)
+[![CI](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml/badge.svg)](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/JB-Analytica/model2data/branch/main/graph/badge.svg)](https://codecov.io/gh/JB-Analytica/model2data)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+`model2data` turns **data models into analytics-ready datasets** in seconds.  
+
+Given a **DBML file**, it generates synthetic but realistic data, a complete dbt project scaffold, and everything you need to start analyzing or testing data pipelines.
+
+---
+
+## What problem does it solve?
+
+Building analytics or testing dbt pipelines often requires realistic data, but using real data raises privacy concerns, and creating mock data manually is time-consuming. `model2data` automates this by generating synthetic datasets from your data model definitions, ensuring privacy-safe, deterministic, and relationship-preserving data for development and testing.
+
+---
+
+## How it works (high level)
+
+1. **Parse DBML**: Reads your database schema from a DBML file, extracting tables, columns, types, and relationships.
+2. **Generate Data**: Uses Faker and custom logic to create realistic synthetic data, respecting foreign keys and constraints.
+3. **Scaffold dbt Project**: Creates a dbt project with seeds (CSV files), staging models, profiles, and tests, ready to run with DuckDB.
+
+---
+
+## Installation
+
+```bash
+pip install model2data
+```
+
+---
+
+## Quick start
+
+We provide an example Hacker News dataset in `examples/hackernews.dbml`.
+
+Generate a project with synthetic data:
+
+```bash
+model2data generate --file examples/hackernews.dbml --rows 200 --seed 42
+```
+
+This creates a `dbt_hackernews/` folder with your data and dbt setup.
+
+Run dbt to load and transform the data:
+
+```bash
+cd dbt_hackernews
+dbt deps
+dbt seed
+dbt run
+```
+
+Your analytics-ready dataset is now in DuckDB!
+
+---
+
+## Generated dbt project structure
+
+The generated dbt project includes:
+
+```
+dbt_{project_name}/
+├── seeds/
+│   └── {project_name}/
+│       ├── table1.csv
+│       └── table2.csv
+├── models/
+│   └── {project_name}/
+│       └── staging/
+│           ├── __sources.yml
+│           ├── stg_table1.sql
+│           ├── stg_table1.yml
+│           └── ...
+├── macros/
+│   └── generate_schema_name.sql
+├── dbt_project.yml
+├── profiles.yml  # DuckDB config
+└── {project_name}.duckdb
+```
+
+- **Seeds**: CSV files with generated synthetic data.
+- **Staging Models**: Basic dbt models that load from seeds.
+- **Sources & Tests**: YAML configs defining sources and basic tests (not_null, unique).
+- **Profiles**: Pre-configured for DuckDB with schema handling.
+
+---
+
+## Design decisions / non-goals
+
+- **DuckDB Default**: Chosen for its zero-config, file-based nature, making it easy to get started without database setup. Other adapters can be configured manually.
+- **dbt Integration**: Leverages dbt's transformation capabilities for a familiar workflow in analytics engineering.
+- **Synthetic Data**: Uses deterministic generation for reproducibility; not intended for production use or as a replacement for real data.
+- **Non-goals**: This is not a data migration tool, ETL pipeline, or real-time data generator. It focuses on static, synthetic datasets for testing and prototyping.
+
+---
+
+## Limitations
+
+- Supports basic DBML features; complex constraints or advanced SQL types may not be fully handled.
+- Synthetic data generation is heuristic-based and may not perfectly mimic real-world distributions or edge cases.
+- Currently optimized for DuckDB; other databases require manual profile adjustments.
+- No support for incremental models or advanced dbt features in generated projects.
+
+---
+
+## Roadmap
+
+- Support for additional database adapters (e.g., Snowflake, BigQuery).
+- Enhanced data type handling and custom generators.
+- Integration with more dbt features like incremental models.
+- Web-based DBML editor and data preview.
+
+---
+
+## Contributing
+
+We welcome contributions!
+
+- Open issues for bugs or feature requests.
+- Submit PRs to add new DBML examples, custom data generators, or improvements.
+- Ensure all new features include tests if possible.
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
+
+## Code of Conduct
+
+Please read our [Code of Conduct](CODE_OF_CONDUCT.md) to understand our community standards.
+
+---
+
+## License
+
+MIT License. See LICENSE for details.

model2data-0.1.0/README.md

@@ -0,0 +1,137 @@
+# model2data
+
+[![PyPI](https://img.shields.io/pypi/v/model2data)](https://pypi.org/project/model2data/)
+[![CI](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml/badge.svg)](https://github.com/JB-Analytica/model2data/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/JB-Analytica/model2data/branch/main/graph/badge.svg)](https://codecov.io/gh/JB-Analytica/model2data)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+`model2data` turns **data models into analytics-ready datasets** in seconds.  
+
+Given a **DBML file**, it generates synthetic but realistic data, a complete dbt project scaffold, and everything you need to start analyzing or testing data pipelines.
+
+---
+
+## What problem does it solve?
+
+Building analytics or testing dbt pipelines often requires realistic data, but using real data raises privacy concerns, and creating mock data manually is time-consuming. `model2data` automates this by generating synthetic datasets from your data model definitions, ensuring privacy-safe, deterministic, and relationship-preserving data for development and testing.
+
+---
+
+## How it works (high level)
+
+1. **Parse DBML**: Reads your database schema from a DBML file, extracting tables, columns, types, and relationships.
+2. **Generate Data**: Uses Faker and custom logic to create realistic synthetic data, respecting foreign keys and constraints.
+3. **Scaffold dbt Project**: Creates a dbt project with seeds (CSV files), staging models, profiles, and tests, ready to run with DuckDB.
+
+---
+
+## Installation
+
+```bash
+pip install model2data
+```
+
+---
+
+## Quick start
+
+We provide an example Hacker News dataset in `examples/hackernews.dbml`.
+
+Generate a project with synthetic data:
+
+```bash
+model2data generate --file examples/hackernews.dbml --rows 200 --seed 42
+```
+
+This creates a `dbt_hackernews/` folder with your data and dbt setup.
+
+Run dbt to load and transform the data:
+
+```bash
+cd dbt_hackernews
+dbt deps
+dbt seed
+dbt run
+```
+
+Your analytics-ready dataset is now in DuckDB!
+
+---
+
+## Generated dbt project structure
+
+The generated dbt project includes:
+
+```
+dbt_{project_name}/
+├── seeds/
+│   └── {project_name}/
+│       ├── table1.csv
+│       └── table2.csv
+├── models/
+│   └── {project_name}/
+│       └── staging/
+│           ├── __sources.yml
+│           ├── stg_table1.sql
+│           ├── stg_table1.yml
+│           └── ...
+├── macros/
+│   └── generate_schema_name.sql
+├── dbt_project.yml
+├── profiles.yml  # DuckDB config
+└── {project_name}.duckdb
+```
+
+- **Seeds**: CSV files with generated synthetic data.
+- **Staging Models**: Basic dbt models that load from seeds.
+- **Sources & Tests**: YAML configs defining sources and basic tests (not_null, unique).
+- **Profiles**: Pre-configured for DuckDB with schema handling.
+
+---
+
+## Design decisions / non-goals
+
+- **DuckDB Default**: Chosen for its zero-config, file-based nature, making it easy to get started without database setup. Other adapters can be configured manually.
+- **dbt Integration**: Leverages dbt's transformation capabilities for a familiar workflow in analytics engineering.
+- **Synthetic Data**: Uses deterministic generation for reproducibility; not intended for production use or as a replacement for real data.
+- **Non-goals**: This is not a data migration tool, ETL pipeline, or real-time data generator. It focuses on static, synthetic datasets for testing and prototyping.
+
+---
+
+## Limitations
+
+- Supports basic DBML features; complex constraints or advanced SQL types may not be fully handled.
+- Synthetic data generation is heuristic-based and may not perfectly mimic real-world distributions or edge cases.
+- Currently optimized for DuckDB; other databases require manual profile adjustments.
+- No support for incremental models or advanced dbt features in generated projects.
+
+---
+
+## Roadmap
+
+- Support for additional database adapters (e.g., Snowflake, BigQuery).
+- Enhanced data type handling and custom generators.
+- Integration with more dbt features like incremental models.
+- Web-based DBML editor and data preview.
+
+---
+
+## Contributing
+
+We welcome contributions!
+
+- Open issues for bugs or feature requests.
+- Submit PRs to add new DBML examples, custom data generators, or improvements.
+- Ensure all new features include tests if possible.
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
+
+## Code of Conduct
+
+Please read our [Code of Conduct](CODE_OF_CONDUCT.md) to understand our community standards.
+
+---
+
+## License
+
+MIT License. See LICENSE for details.

model2data-0.1.0/model2data/cli.py

@@ -0,0 +1,160 @@
+from pathlib import Path
+from typing import Optional
+import random
+import shutil
+
+import typer
+from faker import Faker
+
+from model2data.parse.dbml import parse_dbml
+from model2data.generate.core import generate_data_from_dbml
+from model2data.dbt.project import (
+    create_project_scaffold,
+    create_profiles_yml,
+    create_staging_models,
+)
+from model2data.dbt.tests import generate_dbt_yml
+from model2data.utils import normalize_identifier
+
+app = typer.Typer(
+    help=(
+        "model2data: Generate analytics-ready datasets from DBML models.\n\n"
+        "Given a DBML file, this tool produces:\n"
+        "• Synthetic but realistic data\n"
+        "• A runnable dbt project scaffold\n"
+        "• dbt seeds, staging models, and profiles\n"
+    ),
+    add_completion=False,
+)
+
+
+@app.command(help="Generate synthetic data and a dbt project from a DBML model.")
+def main(
+    file: Path = typer.Option(
+        ...,
+        "--file",
+        "-f",
+        exists=True,
+        file_okay=True,
+        dir_okay=False,
+        readable=True,
+        resolve_path=True,
+        help="Path to the DBML file to generate data from.",
+    ),
+    rows: int = typer.Option(
+        100,
+        "--rows",
+        "-r",
+        min=10,
+        help="Number of rows to generate per table.",
+    ),
+    seed: Optional[int] = typer.Option(
+        None,
+        "--seed",
+        help=(
+            "Optional random seed for deterministic generation.\n"
+            "Using the same seed will always produce identical datasets."
+        ),
+    ),
+    name: Optional[str] = typer.Option(
+        None,
+        "--name",
+        "-n",
+        help="Optional override for the generated dbt project's name.",
+    ),
+    force: bool = typer.Option(
+        False,
+        "--force",
+        help="Overwrite the destination directory if it already exists.",
+    ),
+):
+    """
+    Generate synthetic data and a dbt project from a DBML model.
+    """
+
+    # -------------------------
+    # Deterministic seed
+    # -------------------------
+    if seed is not None:
+        random.seed(seed)
+        Faker.seed(seed)
+        typer.echo(f"🔁 Using deterministic seed: {seed}")
+
+    # -------------------------
+    # Parse DBML (names untouched)
+    # -------------------------
+    tables, refs = parse_dbml(file)
+    if not tables:
+        typer.echo("❌ No tables found in the provided DBML file.")
+        raise typer.Exit(1)
+
+    # -------------------------
+    # DBML → dbt name mapping
+    # -------------------------
+    dbt_name_map = {
+        table_name: normalize_identifier(table_name)
+        for table_name in tables.keys()
+    }
+
+    project_name = normalize_identifier(name or file.stem)
+    dest = Path.cwd() / f"dbt_{project_name}"
+    profile_name = f"{project_name}_profile"
+
+    if dest.exists():
+        if not force:
+            typer.echo(
+                f"❌ Destination {dest} already exists.\n"
+                "Use --force to overwrite."
+            )
+            raise typer.Exit(1)
+        shutil.rmtree(dest)
+
+    # -------------------------
+    # dbt project scaffold
+    # -------------------------
+    typer.echo(f"📦 Creating dbt project scaffold at {dest}")
+    create_project_scaffold(dest, project_name, profile_name)
+
+    # -------------------------
+    # Generate synthetic data
+    # -------------------------
+    typer.echo("🧮 Generating synthetic datasets from DBML definitions...")
+    generated_tables = generate_data_from_dbml(
+        tables=tables,
+        refs=refs,
+        base_rows=rows,
+        seed=seed,
+    )
+
+    # -------------------------
+    # Write dbt seeds (normalized names)
+    # -------------------------
+    seeds_path = dest / "seeds/raw"
+    for table_key, df in generated_tables.items():
+        csv_path = seeds_path / f"{table_key}.csv"
+        df.to_csv(csv_path, index=False)
+
+    # -------------------------
+    # Build dbt assets
+    # -------------------------
+    typer.echo("🗂️ Building staging models for generated seeds...")
+    create_staging_models(dest, project_name)
+
+    typer.echo("🧪 Generating dbt yml with tests...")
+    generate_dbt_yml(dest, tables, refs, project_name)
+
+    typer.echo("🪪 Ensuring dbt profile exists...")
+    create_profiles_yml(dest, profile_name)
+
+    # Keep original DBML for reference
+    shutil.copy(file, dest / file.name)
+
+    # -------------------------
+    # Done
+    # -------------------------
+    typer.echo("\n🎉 model2data generation complete!\n")
+    typer.echo("Next steps:")
+    typer.echo(f"  cd {dest}")
+    typer.echo("  dbt deps")
+    typer.echo("  dbt seed")
+    typer.echo("  dbt run")

model2data-0.1.0/model2data/dbt/project.py

@@ -0,0 +1,74 @@
+from pathlib import Path
+import shutil
+import jinja2
+
+TEMPLATES_DIR = Path(__file__).parent / "templates"
+
+
+def create_project_scaffold(dest: Path, project_name: str, profile_name: str) -> None:
+    dest.mkdir(parents=True, exist_ok=True)
+
+    # dbt folders
+    (dest / "models" / "staging").mkdir(parents=True, exist_ok=True)
+    (dest / "seeds" / "raw").mkdir(parents=True, exist_ok=True)
+    (dest / "analysis").mkdir(exist_ok=True)
+    (dest / "macros").mkdir(exist_ok=True)
+    (dest / "tests").mkdir(exist_ok=True)
+    (dest / "snapshots").mkdir(exist_ok=True)
+
+    # dbt_project.yml
+    _render_template(
+        template_name="dbt_project.yml.jinja",
+        output_path=dest / "dbt_project.yml",
+        context={"project_name": project_name, "profile_name": profile_name},
+    )
+
+    # Copy over any macros from templates
+    template_macros_dir = Path("model2data/dbt/templates/macros")
+    if template_macros_dir.exists():
+        for macro_file in template_macros_dir.glob("*.sql"):
+            target_file = dest / "macros" / macro_file.name
+            if not target_file.exists():
+                target_file.write_text(macro_file.read_text())
+
+
+def create_staging_models(dest: Path, project_name: str) -> None:
+    """
+    Creates staging models in models/staging/ folder that reference raw seed tables as sources.
+    """
+    seeds_path = dest / "seeds" / "raw"
+    models_path = dest / "models" / "staging"
+    models_path.mkdir(parents=True, exist_ok=True)
+
+    for csv_file in seeds_path.glob("*.csv"):
+        table_name = csv_file.stem  # keep full seed name, e.g., raw_stories
+        model_file = models_path / f"stg_{table_name}.sql"
+
+        if not model_file.exists():
+            sql_content = f"""\
+-- Auto-generated staging model for {table_name}
+select *
+from {{{{ source('raw', '{table_name}') }}}}
+                """
+            model_file.write_text(sql_content)
+
+
+def create_profiles_yml(dest: Path, profile_name: str) -> None:
+    profiles_file = dest / "profiles.yml"
+    if profiles_file.exists():
+        content = profiles_file.read_text()
+        if profile_name in content:
+            return
+    _render_template(
+        template_name="profiles.yml.jinja",
+        output_path=profiles_file,
+        context={"profile_name": profile_name},
+    )
+
+
+def _render_template(template_name: str, output_path: Path, context: dict) -> None:
+    template_path = TEMPLATES_DIR / template_name
+    if not template_path.exists():
+        raise FileNotFoundError(f"Template not found: {template_path}")
+    template = jinja2.Template(template_path.read_text())
+    output_path.write_text(template.render(**context))