dbt-extra 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dbt_extra-0.1.0/.gitignore +24 -0
- dbt_extra-0.1.0/CHANGELOG.md +27 -0
- dbt_extra-0.1.0/CONTRIBUTING.md +63 -0
- dbt_extra-0.1.0/LICENSE +21 -0
- dbt_extra-0.1.0/PKG-INFO +206 -0
- dbt_extra-0.1.0/README.md +188 -0
- dbt_extra-0.1.0/SECURITY.md +41 -0
- dbt_extra-0.1.0/docs/assets/dbt-extra-mark-dark.svg +8 -0
- dbt_extra-0.1.0/docs/assets/dbt-extra-mark-mono.svg +8 -0
- dbt_extra-0.1.0/docs/assets/dbt-extra-mark.svg +8 -0
- dbt_extra-0.1.0/docs/compatibility.md +55 -0
- dbt_extra-0.1.0/docs/concepts.md +34 -0
- dbt_extra-0.1.0/docs/development/architecture.md +37 -0
- dbt_extra-0.1.0/docs/development/launch.md +69 -0
- dbt_extra-0.1.0/docs/development/releasing.md +63 -0
- dbt_extra-0.1.0/docs/development/testing.md +82 -0
- dbt_extra-0.1.0/docs/getting-started.md +100 -0
- dbt_extra-0.1.0/docs/guides/ai-assisted-workflow.md +88 -0
- dbt_extra-0.1.0/docs/guides/catalog-round-trip.md +80 -0
- dbt_extra-0.1.0/docs/guides/existing-dbt-project.md +20 -0
- dbt_extra-0.1.0/docs/guides/inspection-and-optimization.md +129 -0
- dbt_extra-0.1.0/docs/guides/production-workflow.md +56 -0
- dbt_extra-0.1.0/docs/guides/source-optimizer-safety.md +168 -0
- dbt_extra-0.1.0/docs/index.md +37 -0
- dbt_extra-0.1.0/docs/logo.md +27 -0
- dbt_extra-0.1.0/docs/reference/catalogs.md +168 -0
- dbt_extra-0.1.0/docs/reference/cli.md +103 -0
- dbt_extra-0.1.0/docs/reference/configuration.md +80 -0
- dbt_extra-0.1.0/docs/troubleshooting.md +72 -0
- dbt_extra-0.1.0/examples/jaffle_shop/.gitignore +5 -0
- dbt_extra-0.1.0/examples/jaffle_shop/README.md +81 -0
- dbt_extra-0.1.0/examples/jaffle_shop/dbt_project.yml +43 -0
- dbt_extra-0.1.0/examples/jaffle_shop/extra_project.yaml +16 -0
- dbt_extra-0.1.0/examples/jaffle_shop/macros/hash_relation_attributes.sql +15 -0
- dbt_extra-0.1.0/examples/jaffle_shop/macros/nonnegative.sql +5 -0
- dbt_extra-0.1.0/examples/jaffle_shop/macros/round_money.sql +3 -0
- dbt_extra-0.1.0/examples/jaffle_shop/macros/rounded_total.sql +3 -0
- dbt_extra-0.1.0/examples/jaffle_shop/macros/schema.yml +20 -0
- dbt_extra-0.1.0/examples/jaffle_shop/macros/unique_combination.sql +6 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/brz/brz.yml +31 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/brz/brz_customers.sql +7 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/brz/brz_orders.sql +8 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/brz/brz_payments.sql +8 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/brz/sources.yml +79 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/capabilities/capabilities.yml +58 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/capabilities/gld_customer_attribute_hash.sql +6 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/capabilities/gld_customer_cohorts.sql +40 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/capabilities/gld_join_counts.sql +61 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/capabilities/gld_macro_arguments.sql +10 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/customer_revenue.sql +9 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/gld.yml +115 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/gld_customers.sql +24 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/gld_daily_sales.sql +11 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/gld_orders.sql +23 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/gld_payment_method_summary.sql +10 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/macro_revenue.sql +2 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/README.md +139 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_customer_leaderboard.sql +64 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_customer_lifetime_value.sql +20 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_customer_order_ranking.sql +23 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_distinct_paying_customers.sql +20 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_high_value_customers.sql +26 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_boolean_normalization.sql +5 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_constants.sql +5 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_customer_profile.sql +19 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_join_simplification.sql +7 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_predicates.sql +8 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_projections.sql +19 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_schema_projection.sql +5 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_optimizer_shared_subqueries.sql +13 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/gld_payment_union.sql +15 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/gld/optimization/optimization.yml +208 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/schema.yml +37 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/slv/orders.sql +8 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/slv/slv.yml +76 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/slv/slv_customers.sql +7 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/slv/slv_orders.sql +8 -0
- dbt_extra-0.1.0/examples/jaffle_shop/models/slv/slv_payments.sql +8 -0
- dbt_extra-0.1.0/examples/jaffle_shop/profiles.yml +7 -0
- dbt_extra-0.1.0/examples/jaffle_shop/seeds/LICENSE +201 -0
- dbt_extra-0.1.0/examples/jaffle_shop/seeds/README.md +26 -0
- dbt_extra-0.1.0/examples/jaffle_shop/seeds/demo_raw_orders.csv +5 -0
- dbt_extra-0.1.0/examples/jaffle_shop/seeds/raw_customers.csv +101 -0
- dbt_extra-0.1.0/examples/jaffle_shop/seeds/raw_orders.csv +100 -0
- dbt_extra-0.1.0/examples/jaffle_shop/seeds/raw_payments.csv +114 -0
- dbt_extra-0.1.0/examples/jaffle_shop/snapshots/slv/customer_history.sql +5 -0
- dbt_extra-0.1.0/examples/jaffle_shop/tests/demo_revenue_reconciliation.sql +27 -0
- dbt_extra-0.1.0/examples/jaffle_shop/tests/jaffle_shop_reconciliation.sql +27 -0
- dbt_extra-0.1.0/pyproject.toml +44 -0
- dbt_extra-0.1.0/scripts/smoke_distribution.py +116 -0
- dbt_extra-0.1.0/src/dbt_extra/__init__.py +18 -0
- dbt_extra-0.1.0/src/dbt_extra/__main__.py +3 -0
- dbt_extra-0.1.0/src/dbt_extra/analysis/__init__.py +5 -0
- dbt_extra-0.1.0/src/dbt_extra/analysis/analyzer.py +111 -0
- dbt_extra-0.1.0/src/dbt_extra/analysis/dependencies.py +26 -0
- dbt_extra-0.1.0/src/dbt_extra/analysis/lineage.py +35 -0
- dbt_extra-0.1.0/src/dbt_extra/analysis/models.py +34 -0
- dbt_extra-0.1.0/src/dbt_extra/analysis/types.py +12 -0
- dbt_extra-0.1.0/src/dbt_extra/cli.py +243 -0
- dbt_extra-0.1.0/src/dbt_extra/config.py +71 -0
- dbt_extra-0.1.0/src/dbt_extra/enrichment.py +84 -0
- dbt_extra-0.1.0/src/dbt_extra/errors.py +5 -0
- dbt_extra-0.1.0/src/dbt_extra/files/__init__.py +1 -0
- dbt_extra-0.1.0/src/dbt_extra/files/changes.py +185 -0
- dbt_extra-0.1.0/src/dbt_extra/files/paths.py +29 -0
- dbt_extra-0.1.0/src/dbt_extra/files/yaml.py +150 -0
- dbt_extra-0.1.0/src/dbt_extra/inspection.py +188 -0
- dbt_extra-0.1.0/src/dbt_extra/optimization.py +160 -0
- dbt_extra-0.1.0/src/dbt_extra/project/__init__.py +6 -0
- dbt_extra-0.1.0/src/dbt_extra/project/artifacts.py +28 -0
- dbt_extra-0.1.0/src/dbt_extra/project/discovery.py +110 -0
- dbt_extra-0.1.0/src/dbt_extra/project/project.py +96 -0
- dbt_extra-0.1.0/src/dbt_extra/project/properties.py +42 -0
- dbt_extra-0.1.0/src/dbt_extra/project/resources.py +42 -0
- dbt_extra-0.1.0/src/dbt_extra/project/selectors.py +30 -0
- dbt_extra-0.1.0/src/dbt_extra/review/__init__.py +6 -0
- dbt_extra-0.1.0/src/dbt_extra/review/contract.py +51 -0
- dbt_extra-0.1.0/src/dbt_extra/review/csv.py +39 -0
- dbt_extra-0.1.0/src/dbt_extra/review/export.py +75 -0
- dbt_extra-0.1.0/src/dbt_extra/review/import_.py +172 -0
- dbt_extra-0.1.0/src/dbt_extra/review/json.py +7 -0
- dbt_extra-0.1.0/src/dbt_extra/review/workbook.py +596 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/__init__.py +6 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/inspection_rules.py +92 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/jinja.py +77 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/macros.py +304 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/parser.py +130 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/qualification.py +143 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/references.py +31 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/relations.py +105 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/renderer.py +75 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/schema_evidence.py +90 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/scopes.py +109 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/shorthand.py +197 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/source_expressions.py +116 -0
- dbt_extra-0.1.0/src/dbt_extra/sql/source_rules.py +317 -0
- dbt_extra-0.1.0/src/dbt_extra/validation.py +26 -0
- dbt_extra-0.1.0/tests/__init__.py +1 -0
- dbt_extra-0.1.0/tests/analysis/test_dependencies.py +20 -0
- dbt_extra-0.1.0/tests/conftest.py +3 -0
- dbt_extra-0.1.0/tests/files/test_changes.py +100 -0
- dbt_extra-0.1.0/tests/files/test_yaml.py +107 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_customer_leaderboard.sql +56 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_customer_lifetime_value.sql +20 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_customer_order_ranking.sql +12 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_distinct_paying_customers.sql +12 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_high_value_customers.sql +16 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_boolean_normalization.sql +5 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_constants.sql +3 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_customer_profile.sql +9 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_join_simplification.sql +7 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_predicates.sql +8 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_projections.sql +13 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_schema_projection.sql +3 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_optimizer_shared_subqueries.sql +13 -0
- dbt_extra-0.1.0/tests/fixtures/optimization/gld_payment_union.sql +9 -0
- dbt_extra-0.1.0/tests/fixtures/optimization_inputs.yml +122 -0
- dbt_extra-0.1.0/tests/helpers.py +90 -0
- dbt_extra-0.1.0/tests/project/test_resources.py +40 -0
- dbt_extra-0.1.0/tests/review/test_workbook.py +786 -0
- dbt_extra-0.1.0/tests/sql/test_jinja.py +23 -0
- dbt_extra-0.1.0/tests/sql/test_macros.py +156 -0
- dbt_extra-0.1.0/tests/sql/test_parser.py +49 -0
- dbt_extra-0.1.0/tests/sql/test_shorthand.py +65 -0
- dbt_extra-0.1.0/tests/sql/test_source_optimization.py +786 -0
- dbt_extra-0.1.0/tests/test_architecture.py +42 -0
- dbt_extra-0.1.0/tests/test_branding.py +191 -0
- dbt_extra-0.1.0/tests/test_documentation.py +307 -0
- dbt_extra-0.1.0/tests/test_logo.py +53 -0
- dbt_extra-0.1.0/tests/test_yaml_style.py +55 -0
- dbt_extra-0.1.0/tests/workflows/test_dbt.py +573 -0
- dbt_extra-0.1.0/tests/workflows/test_inspection_compilation.py +220 -0
- dbt_extra-0.1.0/tests/workflows/test_inspection_optimization.py +144 -0
- dbt_extra-0.1.0/tests/workflows/test_inspection_rules.py +205 -0
- dbt_extra-0.1.0/tests/workflows/test_metadata.py +172 -0
- dbt_extra-0.1.0/tests/workflows/test_optimization_examples.py +317 -0
- dbt_extra-0.1.0/uv.lock +1614 -0
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Python-generated files
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[oc]
|
|
4
|
+
build/
|
|
5
|
+
dist/
|
|
6
|
+
wheels/
|
|
7
|
+
*.egg-info
|
|
8
|
+
|
|
9
|
+
# Virtual environments
|
|
10
|
+
.venv
|
|
11
|
+
.venv-package-smoke/
|
|
12
|
+
|
|
13
|
+
# Local dbt artifacts
|
|
14
|
+
**/target/
|
|
15
|
+
**/dbt_packages/
|
|
16
|
+
**/logs/
|
|
17
|
+
**/.user.yml
|
|
18
|
+
*.duckdb
|
|
19
|
+
*.duckdb.wal
|
|
20
|
+
.pytest_cache/
|
|
21
|
+
**/extra/review/
|
|
22
|
+
|
|
23
|
+
# Temporary Excel lock files
|
|
24
|
+
~$*.xlsx
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## Unreleased
|
|
4
|
+
|
|
5
|
+
## 0.1.0 — Initial release
|
|
6
|
+
|
|
7
|
+
First version of **dbt-extra**.
|
|
8
|
+
|
|
9
|
+
- Distribution `dbt-extra`, Python package `dbt_extra`, and CLI `extra`.
|
|
10
|
+
- Native dbt SQL, property YAML, macros and snapshots remain authoritative.
|
|
11
|
+
- Offline validation, full-model and selected-column inspection, logical plans,
|
|
12
|
+
bounded local macro previews, and explicit analysis coverage reasons.
|
|
13
|
+
- Folder-grouped XLSX and CSV reviews, read-only JSON catalogs, and
|
|
14
|
+
baseline-checked edits back to authored files with YAML comment preservation.
|
|
15
|
+
- Conservative source SQL optimization and missing-metadata enrichment;
|
|
16
|
+
`--preview` / `--dry-run` writes nothing.
|
|
17
|
+
- Optional `extra_project.yaml` and `EXTRA_PROJECT_DIR`; review exports default
|
|
18
|
+
to `extra/review/` with format version 1 and no branded format identifiers.
|
|
19
|
+
- Metadata enrichment and review import update existing dbt YAML entries only;
|
|
20
|
+
missing entries are reported and skipped, never created automatically.
|
|
21
|
+
- Documentation, executable Jaffle Shop examples and native dbt integration
|
|
22
|
+
fixture, tests, package smoke checks and cross-platform CI.
|
|
23
|
+
- MIT licensing and third-party Apache 2.0 seed attribution.
|
|
24
|
+
- Adopted the maintainer-supplied Modular joint / Lagoon SVG icon, preserving
|
|
25
|
+
its paths and palette across light/dark variants, plus a monochrome asset.
|
|
26
|
+
- PyPI trusted publishing after CI on `v*` tags or manual dispatch, with
|
|
27
|
+
release-tag/package-version verification for tag runs.
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# Contributing to dbt-extra
|
|
2
|
+
|
|
3
|
+
extra is an offline dbt transformation companion. Changes should keep authored SQL,
|
|
4
|
+
business metadata and review outputs inspectable, with dbt responsible for
|
|
5
|
+
database execution. Read the [concepts](docs/concepts.md) and
|
|
6
|
+
[architecture](docs/development/architecture.md) first.
|
|
7
|
+
|
|
8
|
+
## Set up and test
|
|
9
|
+
|
|
10
|
+
Use Python 3.11+ and [uv](https://docs.astral.sh/uv/), then run from the repository root:
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
uv sync --locked
|
|
14
|
+
uv run pytest
|
|
15
|
+
uv run extra --project-dir examples/jaffle_shop validate
|
|
16
|
+
uv run extra --project-dir examples/jaffle_shop export --preview
|
|
17
|
+
uv run extra --project-dir examples/jaffle_shop optimize --preview
|
|
18
|
+
uv build
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
The suite includes documentation/link checks and DuckDB result-equivalence tests.
|
|
22
|
+
See [testing](docs/development/testing.md) for the executable dbt checks and
|
|
23
|
+
clean-install package smoke tests. Do not run write-mode optimization against
|
|
24
|
+
bundled teaching inputs just to test a change; copy them to a temporary project.
|
|
25
|
+
|
|
26
|
+
## Propose a change
|
|
27
|
+
|
|
28
|
+
- Useful first contributions include a small redacted dialect/macro reproduction,
|
|
29
|
+
a regression test, or a clearer walkthrough. Check existing issues first; if no
|
|
30
|
+
suitable issue exists, describe your proposed scope before a substantial change.
|
|
31
|
+
- For significant behavior/schema changes, open an issue describing the use case
|
|
32
|
+
before building an incompatible API. Small documentation fixes can go straight
|
|
33
|
+
to a pull request.
|
|
34
|
+
- Keep changes focused. Include a minimal reproduction and regression test for
|
|
35
|
+
bugs; include expected inputs/outputs for new features.
|
|
36
|
+
- Preserve output-column order, declared metadata, validation-before-write
|
|
37
|
+
behavior, unchanged-file bytes and managed-path protections.
|
|
38
|
+
- Prefer explicit errors to guessed schema information. No warehouse access may
|
|
39
|
+
be introduced into offline analysis commands without an explicit design change.
|
|
40
|
+
- Update relevant docs and [CHANGELOG](CHANGELOG.md) for user-visible changes.
|
|
41
|
+
Follow [compatibility](docs/compatibility.md) for schema/API changes.
|
|
42
|
+
- Write block-style YAML and plain GitHub-readable Markdown. Link reference docs
|
|
43
|
+
rather than duplicating their full content in the README.
|
|
44
|
+
- Do not commit credentials, local databases, dbt logs/targets or generated demo
|
|
45
|
+
dbt output. The example workbook and seed data are intentional fixtures.
|
|
46
|
+
|
|
47
|
+
In your pull request, state what changed, why, commands tested, limitations and
|
|
48
|
+
any migration required. If AI helped, review its output and verify business
|
|
49
|
+
assumptions yourself; do not submit untested generated logic as verified behavior.
|
|
50
|
+
|
|
51
|
+
## Report bugs and security issues
|
|
52
|
+
|
|
53
|
+
For ordinary bugs, include dbt-extra/Python/OS versions, the exact command, a minimal
|
|
54
|
+
redacted project and expected versus actual behavior. Never post private
|
|
55
|
+
catalogs, credentials or production data. Use the [security policy](SECURITY.md)
|
|
56
|
+
for potential vulnerabilities rather than disclosing exploit details publicly.
|
|
57
|
+
For source checkouts, also include `git rev-parse HEAD`; unreleased code can share
|
|
58
|
+
the published package's version string. The GitHub issue forms collect these details.
|
|
59
|
+
|
|
60
|
+
## License
|
|
61
|
+
|
|
62
|
+
Contributions to dbt-extra code and documentation are under the [MIT license](LICENSE).
|
|
63
|
+
Preserve third-party notices and the separate Jaffle Shop seed attribution.
|
dbt_extra-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 billybillysss
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
dbt_extra-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: dbt-extra
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Offline dbt transformation review, inspection, editing, and optimization
|
|
5
|
+
Project-URL: Repository, https://github.com/billybillysss/dbt-extra
|
|
6
|
+
Project-URL: Documentation, https://github.com/billybillysss/dbt-extra/blob/HEAD/docs/index.md
|
|
7
|
+
Project-URL: Issues, https://github.com/billybillysss/dbt-extra/issues
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Requires-Python: >=3.11
|
|
11
|
+
Requires-Dist: click>=8.1
|
|
12
|
+
Requires-Dist: jinja2>=3.1
|
|
13
|
+
Requires-Dist: openpyxl>=3.1.5
|
|
14
|
+
Requires-Dist: pyyaml>=6.0.3
|
|
15
|
+
Requires-Dist: ruamel-yaml<0.20,>=0.18
|
|
16
|
+
Requires-Dist: sqlglot<30.25,>=30.8
|
|
17
|
+
Description-Content-Type: text/markdown
|
|
18
|
+
|
|
19
|
+
<p align="center">
|
|
20
|
+
<picture>
|
|
21
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/dbt-extra-mark-dark.svg">
|
|
22
|
+
<img src="docs/assets/dbt-extra-mark.svg" alt="dbt-extra — Modular joint icon" width="128" height="128">
|
|
23
|
+
</picture>
|
|
24
|
+
</p>
|
|
25
|
+
|
|
26
|
+
<h1 align="center">dbt-extra — dbt transformation review and inspection</h1>
|
|
27
|
+
|
|
28
|
+
<p align="center">
|
|
29
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-0D9488" alt="License: MIT"></a>
|
|
30
|
+
<a href="pyproject.toml"><img src="https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white" alt="Python 3.11+"></a>
|
|
31
|
+
<a href="https://github.com/billybillysss/dbt-extra/actions/workflows/ci.yml"><img src="https://github.com/billybillysss/dbt-extra/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
|
32
|
+
<a href="docs/index.md"><img src="https://img.shields.io/badge/docs-read-0D9488" alt="Documentation"></a>
|
|
33
|
+
</p>
|
|
34
|
+
|
|
35
|
+
<p align="center">
|
|
36
|
+
<a href="docs/getting-started.md">Get started</a> · <a href="docs/index.md">Documentation</a> · <a href="docs/guides/ai-assisted-workflow.md">AI-assisted workflow</a> · <a href="examples/jaffle_shop/README.md">Native dbt demo</a> · <a href="CONTRIBUTING.md">Contributing</a>
|
|
37
|
+
</p>
|
|
38
|
+
|
|
39
|
+
**Trace a dbt column back to its SQL. Review transformations in Excel.
|
|
40
|
+
Preview edits to the original files—all offline.**
|
|
41
|
+
|
|
42
|
+
Where does `revenue` come from? Which filters change it? Can a reviewer update
|
|
43
|
+
its description without hunting through YAML? dbt-extra helps answer those questions
|
|
44
|
+
using the dbt project you already have. No warehouse credentials, dbt compilation,
|
|
45
|
+
or second model language required.
|
|
46
|
+
|
|
47
|
+
## See where a column comes from
|
|
48
|
+
|
|
49
|
+
Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/). From a terminal:
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
git clone https://github.com/billybillysss/dbt-extra.git
|
|
53
|
+
cd dbt-extra
|
|
54
|
+
uv sync --locked
|
|
55
|
+
uv run --locked extra --project-dir examples/jaffle_shop inspect customer_revenue --columns revenue --compile false
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The bundled example follows `customer_revenue → orders → demo_raw_orders`.
|
|
59
|
+
The explanation retains the **completed-order filter**, **decimal cast** and
|
|
60
|
+
**per-customer grouping**, even though you selected only `revenue`.
|
|
61
|
+
|
|
62
|
+
<details>
|
|
63
|
+
<summary>See the SQL output (CLI comments omitted)</summary>
|
|
64
|
+
|
|
65
|
+
```sql
|
|
66
|
+
SELECT
|
|
67
|
+
"extra_selected_output"."revenue" AS "revenue"
|
|
68
|
+
FROM (
|
|
69
|
+
WITH order_totals AS (
|
|
70
|
+
SELECT
|
|
71
|
+
SUM(CAST("extra_input_2".amount AS DECIMAL(12, 2))) AS revenue
|
|
72
|
+
FROM {{ ref('demo_raw_orders') }} AS "extra_input_2"
|
|
73
|
+
WHERE
|
|
74
|
+
"extra_input_2".status = 'completed'
|
|
75
|
+
GROUP BY
|
|
76
|
+
"extra_input_2".customer_id
|
|
77
|
+
)
|
|
78
|
+
SELECT
|
|
79
|
+
order_totals.revenue + 0 AS revenue
|
|
80
|
+
FROM order_totals AS order_totals
|
|
81
|
+
) AS "extra_selected_output";
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
This is a definition explanation, not executed SQL or proof about stored data.
|
|
85
|
+
The seed remains an explicit boundary. Authored files are unchanged.
|
|
86
|
+
|
|
87
|
+
</details>
|
|
88
|
+
|
|
89
|
+
Stop at an upstream model with `--from orders`, or use `--view plan` for a logical
|
|
90
|
+
tree. [Try the walkthrough →](docs/getting-started.md)
|
|
91
|
+
|
|
92
|
+
> **Private preview · 0.1.0:** install from this checkout with repository access.
|
|
93
|
+
> The executable is `extra`; Python imports use `dbt_extra`.
|
|
94
|
+
|
|
95
|
+
## Pick your workflow
|
|
96
|
+
|
|
97
|
+
| You want to… | Use dbt-extra to… |
|
|
98
|
+
| --- | --- |
|
|
99
|
+
| Understand one output column | Follow supported upstream SQL while retaining joins, filters, grouping and row multiplicity. |
|
|
100
|
+
| Review SQL and metadata together | Export a folder-grouped XLSX workbook or CSV, edit supported cells, and preview diffs back to dbt files. |
|
|
101
|
+
| Review an AI-proposed change | Export read-only JSON with SQL, dependencies, column lineage and explicit coverage reasons for an external assistant. |
|
|
102
|
+
| Clean up authored SQL | Preview conservative rewrites that preserve dbt calls; unsafe or opaque cases are skipped with reasons. |
|
|
103
|
+
| Fill missing documentation | Infer available types and unambiguous passthrough descriptions without overwriting authored metadata. |
|
|
104
|
+
|
|
105
|
+
### Review in Excel, apply through a diff
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
uv run --locked extra --project-dir examples/jaffle_shop export --format xlsx --select customer_revenue
|
|
109
|
+
# Edit supported cells in examples/jaffle_shop/extra/review/review.xlsx, then close Excel.
|
|
110
|
+
uv run --locked extra --project-dir examples/jaffle_shop import examples/jaffle_shop/extra/review/review.xlsx --preview
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The workbook groups `TABLE → STEP / OUTPUT → COLUMN` rows. Supported edits patch
|
|
114
|
+
the original SQL/YAML; stale exports are rejected, YAML comments are preserved,
|
|
115
|
+
and unchanged files keep their bytes. Preview writes nothing. Follow the
|
|
116
|
+
[round-trip walkthrough](docs/guides/catalog-round-trip.md) before applying edits.
|
|
117
|
+
|
|
118
|
+
### Preview SQL cleanup and metadata proposals
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
uv run --locked extra --project-dir examples/jaffle_shop optimize --select customer_revenue --preview
|
|
122
|
+
uv run --locked extra --project-dir examples/jaffle_shop enrich --select customer_revenue --preview
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
For more visible before/after SQL rewrites, try the
|
|
126
|
+
[14 focused optimizer demos](examples/jaffle_shop/models/gld/optimization/README.md).
|
|
127
|
+
Logical cleanup is **not** a measured warehouse speedup.
|
|
128
|
+
|
|
129
|
+
## Use your existing dbt project
|
|
130
|
+
|
|
131
|
+
Point `--project-dir` at your project and supply its actual SQL dialect:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
uv run --locked extra --project-dir path/to/your/dbt_project --dialect snowflake inspect your_model --columns your_column
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
dbt SQL and property YAML remain the source of truth. No initialization is
|
|
138
|
+
required. Optional `extra_project.yaml` stores tool settings only:
|
|
139
|
+
|
|
140
|
+
```yaml
|
|
141
|
+
version: 2
|
|
142
|
+
analysis:
|
|
143
|
+
dialect: duckdb
|
|
144
|
+
export:
|
|
145
|
+
format: xlsx
|
|
146
|
+
output_dir: extra/review
|
|
147
|
+
select:
|
|
148
|
+
- 'path:models/gld/**'
|
|
149
|
+
optimization:
|
|
150
|
+
format: pretty
|
|
151
|
+
preserve_macros: true
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Explicit CLI values replace configured values, including lists. `--preview`
|
|
155
|
+
(`--dry-run`) writes no files. Without it, writing commands apply their
|
|
156
|
+
[documented writes](docs/reference/cli.md). See the
|
|
157
|
+
[existing-project guide](docs/guides/existing-dbt-project.md) for artifact and dialect setup.
|
|
158
|
+
|
|
159
|
+
## Python API
|
|
160
|
+
|
|
161
|
+
```python
|
|
162
|
+
from dbt_extra import Project
|
|
163
|
+
from dbt_extra.analysis import catalog
|
|
164
|
+
from dbt_extra.optimization import optimize
|
|
165
|
+
|
|
166
|
+
project = Project("examples/jaffle_shop")
|
|
167
|
+
document = catalog(project, project.select(["customer_revenue"]))
|
|
168
|
+
changes, skipped = optimize(project, select=["customer_revenue"])
|
|
169
|
+
print(changes.diffs())
|
|
170
|
+
# changes.apply() # Explicitly accept the proposed source edits.
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
The CLI is the preferred entry point; Python integrations should pin and test
|
|
174
|
+
versions. See [architecture](docs/development/architecture.md) for module boundaries.
|
|
175
|
+
|
|
176
|
+
## Safety and maturity
|
|
177
|
+
|
|
178
|
+
extra is pre-1.0 and conservative by design:
|
|
179
|
+
|
|
180
|
+
- **Unknowns stay explicit.** Coverage depends on dialect, schemas and Jinja.
|
|
181
|
+
Bounded static local macros can be previewed; runtime/adapter-dependent macros
|
|
182
|
+
remain opaque. Generated SQL never replaces an authored macro call.
|
|
183
|
+
- **dbt still owns execution.** dbt-extra is not a dbt engine, metrics service or
|
|
184
|
+
warehouse optimizer. Inspection explains definitions, not equivalence to
|
|
185
|
+
persisted or incremental relations. Run your own dbt parsing, tests and builds.
|
|
186
|
+
- **Writes still need review.** Package/artifact-only resources and inherited
|
|
187
|
+
version properties are read-only. Full opaque-template edits are structure-checked,
|
|
188
|
+
not dbt-compiled. Baseline checks and rollback attempts are not crash-safe
|
|
189
|
+
multi-file transactions or file locks.
|
|
190
|
+
|
|
191
|
+
See [compatibility](docs/compatibility.md) and the
|
|
192
|
+
[production adoption checklist](docs/guides/production-workflow.md).
|
|
193
|
+
|
|
194
|
+
## Help shape dbt-extra
|
|
195
|
+
|
|
196
|
+
Try one model from your own project. If analysis stops, share a **minimal redacted
|
|
197
|
+
reproduction** and the coverage reason in an
|
|
198
|
+
[issue](https://github.com/billybillysss/dbt-extra/issues). Dialect edge cases,
|
|
199
|
+
documentation improvements and regression tests are especially useful.
|
|
200
|
+
Read [Contributing](CONTRIBUTING.md) to get started; report vulnerabilities through
|
|
201
|
+
the [security policy](SECURITY.md), not a public bug report.
|
|
202
|
+
|
|
203
|
+
[Get started](docs/getting-started.md) · [CLI](docs/reference/cli.md) ·
|
|
204
|
+
[Configuration](docs/reference/configuration.md) · [Review contract](docs/reference/catalogs.md) ·
|
|
205
|
+
[Inspection and optimization](docs/guides/inspection-and-optimization.md) ·
|
|
206
|
+
[Native example](examples/jaffle_shop/README.md) · [Contributing](CONTRIBUTING.md)
|
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<picture>
|
|
3
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/dbt-extra-mark-dark.svg">
|
|
4
|
+
<img src="docs/assets/dbt-extra-mark.svg" alt="dbt-extra — Modular joint icon" width="128" height="128">
|
|
5
|
+
</picture>
|
|
6
|
+
</p>
|
|
7
|
+
|
|
8
|
+
<h1 align="center">dbt-extra — dbt transformation review and inspection</h1>
|
|
9
|
+
|
|
10
|
+
<p align="center">
|
|
11
|
+
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-0D9488" alt="License: MIT"></a>
|
|
12
|
+
<a href="pyproject.toml"><img src="https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white" alt="Python 3.11+"></a>
|
|
13
|
+
<a href="https://github.com/billybillysss/dbt-extra/actions/workflows/ci.yml"><img src="https://github.com/billybillysss/dbt-extra/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
|
14
|
+
<a href="docs/index.md"><img src="https://img.shields.io/badge/docs-read-0D9488" alt="Documentation"></a>
|
|
15
|
+
</p>
|
|
16
|
+
|
|
17
|
+
<p align="center">
|
|
18
|
+
<a href="docs/getting-started.md">Get started</a> · <a href="docs/index.md">Documentation</a> · <a href="docs/guides/ai-assisted-workflow.md">AI-assisted workflow</a> · <a href="examples/jaffle_shop/README.md">Native dbt demo</a> · <a href="CONTRIBUTING.md">Contributing</a>
|
|
19
|
+
</p>
|
|
20
|
+
|
|
21
|
+
**Trace a dbt column back to its SQL. Review transformations in Excel.
|
|
22
|
+
Preview edits to the original files—all offline.**
|
|
23
|
+
|
|
24
|
+
Where does `revenue` come from? Which filters change it? Can a reviewer update
|
|
25
|
+
its description without hunting through YAML? dbt-extra helps answer those questions
|
|
26
|
+
using the dbt project you already have. No warehouse credentials, dbt compilation,
|
|
27
|
+
or second model language required.
|
|
28
|
+
|
|
29
|
+
## See where a column comes from
|
|
30
|
+
|
|
31
|
+
Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/). From a terminal:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
git clone https://github.com/billybillysss/dbt-extra.git
|
|
35
|
+
cd dbt-extra
|
|
36
|
+
uv sync --locked
|
|
37
|
+
uv run --locked extra --project-dir examples/jaffle_shop inspect customer_revenue --columns revenue --compile false
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The bundled example follows `customer_revenue → orders → demo_raw_orders`.
|
|
41
|
+
The explanation retains the **completed-order filter**, **decimal cast** and
|
|
42
|
+
**per-customer grouping**, even though you selected only `revenue`.
|
|
43
|
+
|
|
44
|
+
<details>
|
|
45
|
+
<summary>See the SQL output (CLI comments omitted)</summary>
|
|
46
|
+
|
|
47
|
+
```sql
|
|
48
|
+
SELECT
|
|
49
|
+
"extra_selected_output"."revenue" AS "revenue"
|
|
50
|
+
FROM (
|
|
51
|
+
WITH order_totals AS (
|
|
52
|
+
SELECT
|
|
53
|
+
SUM(CAST("extra_input_2".amount AS DECIMAL(12, 2))) AS revenue
|
|
54
|
+
FROM {{ ref('demo_raw_orders') }} AS "extra_input_2"
|
|
55
|
+
WHERE
|
|
56
|
+
"extra_input_2".status = 'completed'
|
|
57
|
+
GROUP BY
|
|
58
|
+
"extra_input_2".customer_id
|
|
59
|
+
)
|
|
60
|
+
SELECT
|
|
61
|
+
order_totals.revenue + 0 AS revenue
|
|
62
|
+
FROM order_totals AS order_totals
|
|
63
|
+
) AS "extra_selected_output";
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
This is a definition explanation, not executed SQL or proof about stored data.
|
|
67
|
+
The seed remains an explicit boundary. Authored files are unchanged.
|
|
68
|
+
|
|
69
|
+
</details>
|
|
70
|
+
|
|
71
|
+
Stop at an upstream model with `--from orders`, or use `--view plan` for a logical
|
|
72
|
+
tree. [Try the walkthrough →](docs/getting-started.md)
|
|
73
|
+
|
|
74
|
+
> **Private preview · 0.1.0:** install from this checkout with repository access.
|
|
75
|
+
> The executable is `extra`; Python imports use `dbt_extra`.
|
|
76
|
+
|
|
77
|
+
## Pick your workflow
|
|
78
|
+
|
|
79
|
+
| You want to… | Use dbt-extra to… |
|
|
80
|
+
| --- | --- |
|
|
81
|
+
| Understand one output column | Follow supported upstream SQL while retaining joins, filters, grouping and row multiplicity. |
|
|
82
|
+
| Review SQL and metadata together | Export a folder-grouped XLSX workbook or CSV, edit supported cells, and preview diffs back to dbt files. |
|
|
83
|
+
| Review an AI-proposed change | Export read-only JSON with SQL, dependencies, column lineage and explicit coverage reasons for an external assistant. |
|
|
84
|
+
| Clean up authored SQL | Preview conservative rewrites that preserve dbt calls; unsafe or opaque cases are skipped with reasons. |
|
|
85
|
+
| Fill missing documentation | Infer available types and unambiguous passthrough descriptions without overwriting authored metadata. |
|
|
86
|
+
|
|
87
|
+
### Review in Excel, apply through a diff
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
uv run --locked extra --project-dir examples/jaffle_shop export --format xlsx --select customer_revenue
|
|
91
|
+
# Edit supported cells in examples/jaffle_shop/extra/review/review.xlsx, then close Excel.
|
|
92
|
+
uv run --locked extra --project-dir examples/jaffle_shop import examples/jaffle_shop/extra/review/review.xlsx --preview
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The workbook groups `TABLE → STEP / OUTPUT → COLUMN` rows. Supported edits patch
|
|
96
|
+
the original SQL/YAML; stale exports are rejected, YAML comments are preserved,
|
|
97
|
+
and unchanged files keep their bytes. Preview writes nothing. Follow the
|
|
98
|
+
[round-trip walkthrough](docs/guides/catalog-round-trip.md) before applying edits.
|
|
99
|
+
|
|
100
|
+
### Preview SQL cleanup and metadata proposals
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
uv run --locked extra --project-dir examples/jaffle_shop optimize --select customer_revenue --preview
|
|
104
|
+
uv run --locked extra --project-dir examples/jaffle_shop enrich --select customer_revenue --preview
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
For more visible before/after SQL rewrites, try the
|
|
108
|
+
[14 focused optimizer demos](examples/jaffle_shop/models/gld/optimization/README.md).
|
|
109
|
+
Logical cleanup is **not** a measured warehouse speedup.
|
|
110
|
+
|
|
111
|
+
## Use your existing dbt project
|
|
112
|
+
|
|
113
|
+
Point `--project-dir` at your project and supply its actual SQL dialect:
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
uv run --locked extra --project-dir path/to/your/dbt_project --dialect snowflake inspect your_model --columns your_column
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
dbt SQL and property YAML remain the source of truth. No initialization is
|
|
120
|
+
required. Optional `extra_project.yaml` stores tool settings only:
|
|
121
|
+
|
|
122
|
+
```yaml
|
|
123
|
+
version: 2
|
|
124
|
+
analysis:
|
|
125
|
+
dialect: duckdb
|
|
126
|
+
export:
|
|
127
|
+
format: xlsx
|
|
128
|
+
output_dir: extra/review
|
|
129
|
+
select:
|
|
130
|
+
- 'path:models/gld/**'
|
|
131
|
+
optimization:
|
|
132
|
+
format: pretty
|
|
133
|
+
preserve_macros: true
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
Explicit CLI values replace configured values, including lists. `--preview`
|
|
137
|
+
(`--dry-run`) writes no files. Without it, writing commands apply their
|
|
138
|
+
[documented writes](docs/reference/cli.md). See the
|
|
139
|
+
[existing-project guide](docs/guides/existing-dbt-project.md) for artifact and dialect setup.
|
|
140
|
+
|
|
141
|
+
## Python API
|
|
142
|
+
|
|
143
|
+
```python
|
|
144
|
+
from dbt_extra import Project
|
|
145
|
+
from dbt_extra.analysis import catalog
|
|
146
|
+
from dbt_extra.optimization import optimize
|
|
147
|
+
|
|
148
|
+
project = Project("examples/jaffle_shop")
|
|
149
|
+
document = catalog(project, project.select(["customer_revenue"]))
|
|
150
|
+
changes, skipped = optimize(project, select=["customer_revenue"])
|
|
151
|
+
print(changes.diffs())
|
|
152
|
+
# changes.apply() # Explicitly accept the proposed source edits.
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
The CLI is the preferred entry point; Python integrations should pin and test
|
|
156
|
+
versions. See [architecture](docs/development/architecture.md) for module boundaries.
|
|
157
|
+
|
|
158
|
+
## Safety and maturity
|
|
159
|
+
|
|
160
|
+
extra is pre-1.0 and conservative by design:
|
|
161
|
+
|
|
162
|
+
- **Unknowns stay explicit.** Coverage depends on dialect, schemas and Jinja.
|
|
163
|
+
Bounded static local macros can be previewed; runtime/adapter-dependent macros
|
|
164
|
+
remain opaque. Generated SQL never replaces an authored macro call.
|
|
165
|
+
- **dbt still owns execution.** dbt-extra is not a dbt engine, metrics service or
|
|
166
|
+
warehouse optimizer. Inspection explains definitions, not equivalence to
|
|
167
|
+
persisted or incremental relations. Run your own dbt parsing, tests and builds.
|
|
168
|
+
- **Writes still need review.** Package/artifact-only resources and inherited
|
|
169
|
+
version properties are read-only. Full opaque-template edits are structure-checked,
|
|
170
|
+
not dbt-compiled. Baseline checks and rollback attempts are not crash-safe
|
|
171
|
+
multi-file transactions or file locks.
|
|
172
|
+
|
|
173
|
+
See [compatibility](docs/compatibility.md) and the
|
|
174
|
+
[production adoption checklist](docs/guides/production-workflow.md).
|
|
175
|
+
|
|
176
|
+
## Help shape dbt-extra
|
|
177
|
+
|
|
178
|
+
Try one model from your own project. If analysis stops, share a **minimal redacted
|
|
179
|
+
reproduction** and the coverage reason in an
|
|
180
|
+
[issue](https://github.com/billybillysss/dbt-extra/issues). Dialect edge cases,
|
|
181
|
+
documentation improvements and regression tests are especially useful.
|
|
182
|
+
Read [Contributing](CONTRIBUTING.md) to get started; report vulnerabilities through
|
|
183
|
+
the [security policy](SECURITY.md), not a public bug report.
|
|
184
|
+
|
|
185
|
+
[Get started](docs/getting-started.md) · [CLI](docs/reference/cli.md) ·
|
|
186
|
+
[Configuration](docs/reference/configuration.md) · [Review contract](docs/reference/catalogs.md) ·
|
|
187
|
+
[Inspection and optimization](docs/guides/inspection-and-optimization.md) ·
|
|
188
|
+
[Native example](examples/jaffle_shop/README.md) · [Contributing](CONTRIBUTING.md)
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
## Support scope
|
|
4
|
+
|
|
5
|
+
extra is pre-1.0. There is no published LTS branch, guaranteed response time or
|
|
6
|
+
security SLA. Evaluate the latest code and dependency updates, pin versions, and
|
|
7
|
+
test upgrades in your own environment. Do not infer production certification from
|
|
8
|
+
offline validation or a passing demo.
|
|
9
|
+
|
|
10
|
+
## Report a vulnerability
|
|
11
|
+
|
|
12
|
+
Do **not** open a public issue with exploit details, credentials, proprietary
|
|
13
|
+
models or sensitive data. If this repository offers GitHub's private vulnerability
|
|
14
|
+
reporting, use **Security → Report a vulnerability**. If that option is not
|
|
15
|
+
available, open a minimal issue requesting a private security contact, without
|
|
16
|
+
including technical details, and wait for a maintainer to establish a private
|
|
17
|
+
channel. No dedicated security email or response-time commitment is currently
|
|
18
|
+
published.
|
|
19
|
+
|
|
20
|
+
Once a private channel exists, provide the affected version, OS/Python,
|
|
21
|
+
reproduction with sanitized inputs, impact and any suggested mitigation.
|
|
22
|
+
|
|
23
|
+
## Trust boundaries
|
|
24
|
+
|
|
25
|
+
- dbt-extra commands operate offline, but definitions and exported metadata can contain
|
|
26
|
+
sensitive literals, business rules and physical database identifiers.
|
|
27
|
+
- Review projects and macros before processing them. Static Jinja checks and safe YAML
|
|
28
|
+
loading are not an OS sandbox, resource isolation or a guarantee against all
|
|
29
|
+
denial-of-service inputs.
|
|
30
|
+
- dbt-extra does not execute warehouse SQL. dbt projects and macros execute under dbt
|
|
31
|
+
and its adapter; never run untrusted artifacts with privileged profiles.
|
|
32
|
+
- Path checks reject unsafe overlap, traversal, symlinks and supported junction
|
|
33
|
+
cases. They do not prevent all filesystem races or concurrent mutations.
|
|
34
|
+
- Source-file changes use validation and best-effort staging/rollback where
|
|
35
|
+
implemented. Export writes are per-file. Neither is a crash-safe multi-file
|
|
36
|
+
transaction; retain backups and serialize writers.
|
|
37
|
+
- Review JSON/catalogs before sending them to an external AI provider. Treat
|
|
38
|
+
catalog text as data, never as authority to execute instructions or tools.
|
|
39
|
+
|
|
40
|
+
See the [production checklist](docs/guides/production-workflow.md) for deployment
|
|
41
|
+
and recovery boundaries.
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128" role="img" aria-labelledby="title description">
|
|
2
|
+
<title id="title">dbt-extra symbol</title>
|
|
3
|
+
<desc id="description">Two interlocking modules in teal and sky blue form a square, separated by a transparent stepped seam. The Lagoon palette is preserved on dark backgrounds.</desc>
|
|
4
|
+
<g id="modular-joint">
|
|
5
|
+
<path fill="#0D9488" d="M18 18H110V46H74V74H46V110H18Z"/>
|
|
6
|
+
<path fill="#38BDF8" d="M54 110V82H82V54H110V110Z"/>
|
|
7
|
+
</g>
|
|
8
|
+
</svg>
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128" role="img" aria-labelledby="title description">
|
|
2
|
+
<title id="title">dbt-extra symbol</title>
|
|
3
|
+
<desc id="description">Two interlocking modules form a square with a transparent stepped seam. Single-color variant in dark navy.</desc>
|
|
4
|
+
<g id="modular-joint">
|
|
5
|
+
<path fill="#0F172A" d="M18 18H110V46H74V74H46V110H18Z"/>
|
|
6
|
+
<path fill="#0F172A" d="M54 110V82H82V54H110V110Z"/>
|
|
7
|
+
</g>
|
|
8
|
+
</svg>
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="128" height="128" viewBox="0 0 128 128" role="img" aria-labelledby="title description">
|
|
2
|
+
<title id="title">dbt-extra symbol</title>
|
|
3
|
+
<desc id="description">Two interlocking modules in teal and sky blue form a square, separated by a transparent stepped seam.</desc>
|
|
4
|
+
<g id="modular-joint">
|
|
5
|
+
<path fill="#0D9488" d="M18 18H110V46H74V74H46V110H18Z"/>
|
|
6
|
+
<path fill="#38BDF8" d="M54 110V82H82V54H110V110Z"/>
|
|
7
|
+
</g>
|
|
8
|
+
</svg>
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Compatibility and versioning
|
|
2
|
+
|
|
3
|
+
## Current boundaries
|
|
4
|
+
|
|
5
|
+
| Surface | Current contract |
|
|
6
|
+
| --- | --- |
|
|
7
|
+
| Package | `dbt-extra`, version `0.1.0`; private preview, pre-1.0. Executable `extra`, Python namespace `dbt_extra`. Pin and test upgrades. |
|
|
8
|
+
| Python | Metadata requires 3.11+. CI targets 3.11 and 3.13 on Linux, plus 3.13 on Windows. Other Python/platform combinations are not thereby certified. |
|
|
9
|
+
| SQLGlot | Runtime dependency is constrained to `>=30.8,<30.25`, tested at 30.8.0 and 30.24.0. Dialect availability does not establish warehouse execution support. |
|
|
10
|
+
| Tool settings | Optional `extra_project.yaml`, `version: 2`; no transformation authoring schema. |
|
|
11
|
+
| Review JSON | `format_version: 1`, `source_of_truth: dbt`. Read-only analysis; check coverage and the envelope. |
|
|
12
|
+
| dbt | Native authored SQL/properties/macros/snapshots. No compile/runtime emulation. Execution compatibility depends on your dbt and adapter versions. The native demo uses DuckDB. |
|
|
13
|
+
| Python internals | Not a separately guaranteed stable public API. Prefer the CLI or explicitly version-tested integrations. |
|
|
14
|
+
|
|
15
|
+
CI declares test targets; a new matrix entry is not proof it has passed until
|
|
16
|
+
that workflow runs successfully. The development environment uses DuckDB for
|
|
17
|
+
local execution checks; it does not run a remote warehouse integration.
|
|
18
|
+
No macOS CI or warehouse performance benchmark is currently provided.
|
|
19
|
+
|
|
20
|
+
The supported range stops before 30.25: 30.25.0 failed existing window-alias
|
|
21
|
+
`QUALIFY` regressions, and fresh-install checks with 30.29.0 exposed further API
|
|
22
|
+
and SQL-rendering differences. The exclusion is not a general certification of
|
|
23
|
+
every other SQLGlot version; CI also tests a freshly resolved installed wheel.
|
|
24
|
+
|
|
25
|
+
## Initial version
|
|
26
|
+
|
|
27
|
+
`dbt-extra` starts at `0.1.0` as a private preview.
|
|
28
|
+
Install from an authorized source checkout and record its Git commit when
|
|
29
|
+
reporting results. The repository stays private until
|
|
30
|
+
verification and separate maintainer approval to make it public.
|
|
31
|
+
|
|
32
|
+
Public visibility, package publication and branding are separate release gates;
|
|
33
|
+
see the [release checklist](development/releasing.md).
|
|
34
|
+
|
|
35
|
+
## Upgrade policy
|
|
36
|
+
|
|
37
|
+
Before upgrading, keep a Git baseline and export reviews into an isolated
|
|
38
|
+
directory. Compare validation, SQL, properties, review JSON, mutation diffs and
|
|
39
|
+
dbt results. Check [CHANGELOG](../CHANGELOG.md) for documented changes.
|
|
40
|
+
|
|
41
|
+
Use the responsibility-based packages described in [architecture](development/architecture.md)
|
|
42
|
+
for version-tested Python integrations. Review files must follow the
|
|
43
|
+
[review contract](reference/catalogs.md), including their export baseline.
|
|
44
|
+
|
|
45
|
+
Use `optimization.preserve_macros: true`; native rewrites never expand macros.
|
|
46
|
+
`--preview` is the unified no-write option; `--dry-run` remains an alias.
|
|
47
|
+
|
|
48
|
+
## Maintainer policy for future changes
|
|
49
|
+
|
|
50
|
+
Record user-visible changes under Unreleased. Document CLI/config/schema changes,
|
|
51
|
+
add regression tests, and provide migration instructions for incompatible changes.
|
|
52
|
+
If the JSON contract becomes incompatible, update its format version rather than
|
|
53
|
+
silently reinterpreting existing fields. Additive fields should not require
|
|
54
|
+
consumers to reject an otherwise supported envelope. dbt-extra does not currently
|
|
55
|
+
provide a general JSON Schema validator or migration service.
|