dbt-dagsterizer 0.1.0__tar.gz

dbt_dagsterizer-0.1.0/.gitignore

@@ -0,0 +1,36 @@
+__pycache__/
+*.py[cod]
+*$py.class
+
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+.coverage.*
+htmlcov/
+coverage.xml
+pytestdebug.log
+
+.tox/
+.nox/
+
+.venv/
+venv/
+ENV/
+env/
+
+build/
+dist/
+*.egg-info/
+.eggs/
+*.egg
+
+.env
+.env.*
+!.env.example
+
+.DS_Store
+.idea/
+.vscode/
+*.iml
+

dbt_dagsterizer-0.1.0/PKG-INFO

@@ -0,0 +1,35 @@
+Metadata-Version: 2.4
+Name: dbt-dagsterizer
+Version: 0.1.0
+Summary: Dagster automation driven by dbt metadata (Luban CI)
+Requires-Python: >=3.12
+Requires-Dist: click>=8.3.1
+Requires-Dist: cookiecutter>=2.6.0
+Requires-Dist: dagster-dbt<1,>=0.26
+Requires-Dist: dagster<2,>=1.12
+Requires-Dist: dbt-core<2,>=1.9
+Requires-Dist: dbt-starrocks<2,>=1.9
+Requires-Dist: pymysql>=1.1.0
+Requires-Dist: ruamel-yaml>=0.19.1
+Description-Content-Type: text/markdown
+
+# dbt-dagsterizer
+
+`dbt-dagsterizer` is a Python package that will host the reusable Dagster automation currently embedded in the Dagster+dbt+StarRocks code-location template.
+
+This package is being introduced incrementally. The first step is to provide a stable CLI entrypoint and a minimal, always-loadable Dagster Definitions builder for empty/skeleton projects.
+
+## CLI
+
+```bash
+dbt-dagsterizer --help
+```
+
+## Python
+
+```python
+from dbt_dagsterizer.api import build_definitions
+
+defs = build_definitions(dbt_project_dir="./dbt_project")
+```
+

dbt_dagsterizer-0.1.0/README.md

@@ -0,0 +1,20 @@
+# dbt-dagsterizer
+
+`dbt-dagsterizer` is a Python package that will host the reusable Dagster automation currently embedded in the Dagster+dbt+StarRocks code-location template.
+
+This package is being introduced incrementally. The first step is to provide a stable CLI entrypoint and a minimal, always-loadable Dagster Definitions builder for empty/skeleton projects.
+
+## CLI
+
+```bash
+dbt-dagsterizer --help
+```
+
+## Python
+
+```python
+from dbt_dagsterizer.api import build_definitions
+
+defs = build_definitions(dbt_project_dir="./dbt_project")
+```
+

dbt_dagsterizer-0.1.0/docs/README.md

@@ -0,0 +1,25 @@
+# dbt-dagsterizer docs
+
+This directory contains developer-facing documentation for `dbt-dagsterizer`.
+
+## Start here
+
+- Getting started: `getting-started.md`
+- Concepts overview: `concepts/overview.md`
+- CLI reference: `concepts/cli.md`
+
+## Template docs (Dagster + dbt + StarRocks code location)
+
+- Template overview: `templates/dagster-dbt-starrocks-code-location/README.md`
+- Usage and architecture: `templates/dagster-dbt-starrocks-code-location/template_usage.md`
+- Developer workflow (jobs/schedules/sensors via `dagsterization.yml`): `templates/dagster-dbt-starrocks-code-location/developer_workflow.md`
+- Local development (render + run): `templates/dagster-dbt-starrocks-code-location/local_development.md`
+
+## Developing this repo
+
+- Package development: `development/package-development.md`
+- Codebase tour: `development/codebase-tour.md`
+
+## Legacy paths
+
+Some older docs links may still point under `docs/dagster/**`. Those pages now redirect to the canonical locations above.

dbt_dagsterizer-0.1.0/docs/concepts/cli.md

@@ -0,0 +1,239 @@
+# dbt-dagsterizer CLI
+
+This document describes the `dbt-dagsterizer` CLI.
+
+The CLI exists to:
+
+- Keep the Dagster code location mostly static.
+- Let developers declare orchestration intent close to dbt models, in a reviewable YAML file.
+- Validate that intent early (before Dagster fails at import/runtime).
+
+## Entry points
+
+- `dbt-dagsterizer ...`
+- `python -m dbt_dagsterizer ...`
+
+## Environment
+
+Some dbt configs (e.g. sources) use `env_var(...)`. To make CLI behavior match Dagster runtime behavior, the CLI loads dotenv files before running `dbt parse`:
+
+- `<repo>/.env`
+- `dbt_project/.env`
+
+Existing process environment variables are not overridden.
+
+When `--prepare` is enabled, the CLI may run `dbt parse` to refresh `dbt_project/target/manifest.json`. It writes a sidecar file `dbt_project/target/.luban_manifest_inputs.json` so future runs can detect when a refresh is needed (for example dotenv mtime or dbt target changes).
+
+Current refresh rules when `--prepare` is enabled:
+
+- Manifest missing
+- Dbt target changed
+- Dotenv inputs changed (dotenv paths changed, dotenv removed, or dotenv mtime increased)
+
+## Orchestration file
+
+The CLI reads and writes orchestration intent in a single file:
+
+- `dbt_project/dagsterization.yml`
+
+This file is not a dbt schema YAML, and is intentionally kept outside `dbt_project/models/` so dbt will not parse it.
+
+## Commands
+
+### `project init`
+
+Renders a runnable Dagster + dbt (StarRocks) code-location repo using the embedded cookiecutter template.
+
+```bash
+dbt-dagsterizer project init \
+  --output-dir . \
+  --app-name my_app \
+  --package-name my_app \
+  --project-name "My App" \
+  --author-name "You" \
+  --author-email "you@example.com"
+```
+
+Notes:
+
+- `--output-dir` controls where the project directory is created.
+- `--force` overwrites an existing output directory.
+- By default the generated dbt project is minimal; use `--include-sample-dbt-project` to include sample models.
+
+### `meta validate`
+
+Validates `dagsterization.yml` against `dbt_project/target/manifest.json`.
+
+```bash
+dbt-dagsterizer meta validate --dbt-project-dir dbt_project --prepare
+```
+
+Flags:
+
+- `--prepare/--no-prepare`: when enabled, runs `dbt parse` if the manifest is missing or stale.
+
+### `meta init`
+
+Creates the orchestration YAML file if it does not exist.
+
+```bash
+dbt-dagsterizer meta init --dbt-project-dir dbt_project
+```
+
+Flags:
+
+- `--path`: relative to the dbt project dir (default `dagsterization.yml`)
+- `--force`: overwrite an existing file
+- `--parse`: run `dbt parse` after writing
+
+### `meta job`
+
+Creates/updates a grouped asset job in `dagsterization.yml`.
+
+```bash
+dbt-dagsterizer meta job \
+  --dbt-project-dir dbt_project \
+  --models fact_orders_daily,fact_customer_orders_daily \
+  --name daily_facts_job \
+  --include-upstream \
+  --partitions daily
+```
+
+Selection:
+
+- `--models`: comma-separated model names
+- `--tag`: select all models in the manifest that already have a dbt tag
+
+Flags:
+
+- `--partitions`: `daily|unpartitioned|none`
+- `--prepare`: only used when selecting by `--tag` (needs the manifest)
+- `--parse`: run `dbt parse` after writing
+
+### `meta schedule`
+
+Creates/updates a schedule in `dagsterization.yml`.
+
+```bash
+dbt-dagsterizer meta schedule \
+  --dbt-project-dir dbt_project \
+  --models orders \
+  --name orders_daily_schedule \
+  --hour 2 \
+  --minute 0 \
+  --lookback-days 3 \
+  --enabled
+```
+
+Flags:
+
+- `--parse`: run `dbt parse` after writing
+
+### `meta partition`
+
+Sets partitioning for selected models in `dagsterization.yml`.
+
+```bash
+dbt-dagsterizer meta partition \
+  --dbt-project-dir dbt_project \
+  --models fact_orders_daily,fact_customer_orders_daily \
+  --type daily
+```
+
+Flags:
+
+- `--type`: `daily|unpartitioned|none` (`none` removes the model from `partitions.*`)
+- `--parse`: run `dbt parse` after writing
+
+### `meta asset-job`
+
+Creates/deletes per-model asset jobs for selected models. When present, the derived job name becomes `dbt_<model>_asset_job`.
+
+```bash
+dbt-dagsterizer meta asset-job \
+  --dbt-project-dir dbt_project \
+  --models orders \
+  --no-parse
+```
+
+Flags:
+
+- Adds selected models to `asset_jobs`
+- `--parse`: run `dbt parse` after writing
+
+### `meta asset-job-delete`
+
+Deletes per-model asset jobs for selected models.
+
+```bash
+dbt-dagsterizer meta asset-job-delete \
+  --dbt-project-dir dbt_project \
+  --models orders
+```
+
+Flags:
+
+- `--force`: also remove referencing schedules
+
+### `meta job-delete`
+
+Deletes a grouped job by name.
+
+```bash
+dbt-dagsterizer meta job-delete \
+  --dbt-project-dir dbt_project \
+  --name daily_facts_job
+```
+
+Flags:
+
+- `--force`: also remove references from schedules/propagators
+
+### `meta partition-change detector`
+
+Creates/updates a partition-change detector entry for a model.
+
+```bash
+dbt-dagsterizer meta partition-change detector \
+  --dbt-project-dir dbt_project \
+  --model orders \
+  --enabled \
+  --detect-source ods.orders \
+  --partition-date-expr order_date \
+  --updated-at-expr updated_at \
+  --lookback-days 7 \
+  --offset-days 1
+```
+
+Notes:
+
+- Specify exactly one of `--detect-relation` or `--detect-source`.
+- `--detect-source` uses `source.table` format.
+
+### `meta partition-change propagator`
+
+Creates/updates a partition-change propagation entry for a model.
+
+```bash
+dbt-dagsterizer meta partition-change propagator \
+  --dbt-project-dir dbt_project \
+  --model orders \
+  --enabled \
+  --targets daily_facts_job
+```
+
+### `macros install`
+
+Installs the macro templates shipped with `dbt-dagsterizer` into a dbt project.
+
+```bash
+dbt-dagsterizer macros install --dbt-project-dir dbt_project
+```
+
+Flags:
+
+- `--force`: overwrite files if they exist with different content
+
+## Why `--parse` exists
+
+Dagster reads orchestration intent from the dbt manifest. If you update YAML but do not rebuild `target/manifest.json`, Dagster will not see the change. `--parse` updates the manifest immediately.

dbt_dagsterizer-0.1.0/docs/concepts/dagster-dbt-code-location-template.md

@@ -0,0 +1,122 @@
+# Dagster + dbt (StarRocks) Code Location Template
+
+For the end-to-end developer workflow (template + local validation), start with:
+
+- `../templates/dagster-dbt-starrocks-code-location/README.md`
+
+Dagster deployments commonly use two layers:
+
+- **Dagster Platform**: Webserver + Daemon + Dagster instance storage.
+- **Dagster Code Location**: User code served by `dagster code-server` (gRPC).
+
+This document describes the standard **Dagster + dbt (StarRocks) code location** source template intended for data transformation teams.
+
+## Goal
+
+Provide a runnable, opinionated skeleton that:
+
+- Keeps the Dagster platform and code locations decoupled.
+- Uses **dbt** as the transformation engine (SQL/models/tests).
+- Uses **Dagster** for orchestration, scheduling, observability, and dependency management.
+- Standardizes folder structure and wiring so teams focus on datasets and models.
+
+## When to use this template
+
+Use this template when:
+
+- You want a new Dagster code location that primarily materializes dbt assets.
+- You want consistent conventions across teams (layout, naming, local run, CI expectations).
+
+Do not use this template when:
+
+- You need a Dagster platform.
+- Your pipeline is mostly non-dbt (pure Python ops, Spark jobs, etc.).
+
+## Boundary: Dagster vs dbt
+
+### dbt owns
+
+- Model definitions (SQL in `models/`).
+- Data tests (schema tests, custom tests).
+- Sources/exposures/macros/packages.
+- Model selection semantics (`--select`, tags, state).
+
+### Dagster owns
+
+- Asset orchestration (when/what to run).
+- Observability (asset-level logs, lineage view, run history).
+- Scheduling, sensors, retries, alerting.
+- Coordination with non-dbt assets (ingestion, external tables, ML training).
+
+## Repository layout
+
+The template is a single Git repo representing one code location.
+
+- `src/<package_name>/` Dagster code location module
+  - `definitions.py` exports `defs` (Dagster entrypoint)
+  - `assets/dbt.py` defines how dbt is exposed as Dagster assets
+- `dbt_project/` dbt project
+  - `dbt_project.yml`
+  - `models/` (standard dbt structure)
+  - `profiles.yml` (local-safe default)
+
+## Runtime contract (Luban CI + Dagster)
+
+Luban CI deploys code locations as a Kubernetes Deployment that runs:
+
+`dagster code-server start -h 0.0.0.0 -p <port> -m <package_name>`
+
+This implies:
+
+- The Python module `<package_name>` must import successfully.
+- The module must expose `defs` (via `__init__.py`).
+
+## Local development
+
+For the end-to-end local workflow (render template, start StarRocks, run Dagster), see:
+
+- `../templates/dagster-dbt-starrocks-code-location/local_development.md`
+
+## Production configuration guidance
+
+- Keep secrets out of Git.
+- Prefer configuring warehouse credentials via environment variables and/or mounted Secrets.
+- For production adapters (for example Snowflake, BigQuery, Postgres, StarRocks), update `dbt_project/profiles.yml` to reference credentials via `env_var()`.
+
+## Environments
+
+This template supports configuring the default dbt environment via Cookiecutter (`default_env`).
+
+Common mapping:
+
+- `sandbox`: developer environment and Luban CI `snd` deployment
+- `production`: Luban CI `prd` deployment
+
+Set `DBT_TARGET` to one of the configured environments. In GitOps deployments, you typically keep env var names the same and just provide different values per environment.
+
+## Layers as separate databases (StarRocks)
+
+In many StarRocks deployments, `ods`, `dwd`, and `dws` are separate databases on the same cluster.
+
+The template centralizes layer mapping in `dbt_project.yml` using env-var-driven dbt `vars`.
+
+Convention: ODS source table `name` matches the physical table name. If you need a different physical name, set `identifier` in `dwd/sources.yml`.
+
+## Recommended conventions
+
+- One code location per domain.
+- Shared dimensions as a producer.
+- dbt-first transformations.
+- Stable selection semantics (tags/groups) declared in dbt and consumed by Dagster.
+
+For the supported orchestration config schema (`dbt_project/dagsterization.yml`) and the intended developer workflow, see:
+
+- `../templates/dagster-dbt-starrocks-code-location/developer_workflow.md`
+
+## Extending the skeleton
+
+Common extensions that keep the boundary clean:
+
+- Add ingestion assets in Dagster that produce dbt sources, then make dbt depend on them.
+- Add partitioning in Dagster for incremental models, passing partition ranges into dbt via `--vars`.
+- Add asset checks for dbt tests to show failures at the asset level.

dbt_dagsterizer-0.1.0/docs/concepts/overview.md

@@ -0,0 +1,34 @@
+# dbt-dagsterizer
+
+`dbt-dagsterizer` is the home for reusable Dagster automation driven by dbt metadata.
+
+## Goals
+
+- Keep dbt projects and Dagster code locations decoupled (BYO dbt project).
+- Generate Dagster assets/jobs/schedules/sensors/resources from dbt `manifest.json` plus a small orchestration intent file.
+- Provide a CLI to author and validate orchestration intent with a low learning curve.
+
+## Core concepts
+
+### Manifest-driven
+
+The dbt manifest (`dbt_project/target/manifest.json`) is the stable interface between dbt and Dagster.
+
+### Orchestration intent
+
+Orchestration intent is stored in a single, reviewable YAML file:
+
+- `dbt_project/dagsterization.yml`
+
+This file is not a dbt schema YAML and is intentionally kept outside `dbt_project/models/` so dbt will not parse it.
+
+### Always-loadable definitions
+
+Code locations often need to import successfully even before the dbt project is “real”.
+
+`dbt_dagsterizer.api.build_definitions()` returns a minimal Dagster `Definitions` when there are no dbt models yet, so skeleton repos remain runnable.
+
+## See also
+
+- CLI reference: `cli.md`
+- Code location template concept: `dagster-dbt-code-location-template.md`

dbt_dagsterizer-0.1.0/docs/dagster/concepts/dagster-dbt-code-location-template.md

@@ -0,0 +1,5 @@
+# Legacy path
+
+This page has moved.
+
+Canonical location: `../../concepts/dagster-dbt-code-location-template.md`

dbt_dagsterizer-0.1.0/docs/dagster/concepts/dbt-dagsterizer-cli.md

@@ -0,0 +1,5 @@
+# Legacy path
+
+This page has moved.
+
+Canonical location: `../../concepts/cli.md`

dbt_dagsterizer-0.1.0/docs/dagster/concepts/dbt-dagsterizer.md

@@ -0,0 +1,5 @@
+# Legacy path
+
+This page has moved.
+
+Canonical location: `../../concepts/overview.md`

dbt_dagsterizer-0.1.0/docs/dagster/templates/dagster-dbt-starrocks-code-location/README.md

@@ -0,0 +1,5 @@
+# Legacy path
+
+This directory has moved.
+
+Canonical location: `../../../templates/dagster-dbt-starrocks-code-location/README.md`