brickie-core 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- brickie_core-0.1.0/.github/workflows/ci.yaml +19 -0
- brickie_core-0.1.0/.github/workflows/release.yaml +29 -0
- brickie_core-0.1.0/.gitignore +7 -0
- brickie_core-0.1.0/.pre-commit-config.yaml +27 -0
- brickie_core-0.1.0/.python-version +1 -0
- brickie_core-0.1.0/AGENTS.md +77 -0
- brickie_core-0.1.0/CLAUDE.md +1 -0
- brickie_core-0.1.0/LICENSE +202 -0
- brickie_core-0.1.0/PKG-INFO +90 -0
- brickie_core-0.1.0/README.md +76 -0
- brickie_core-0.1.0/docs/changelog.md +40 -0
- brickie_core-0.1.0/docs/concepts/.gitkeep +0 -0
- brickie_core-0.1.0/docs/concepts/brickie-skills.md +56 -0
- brickie_core-0.1.0/docs/concepts/domains-and-layers.md +89 -0
- brickie_core-0.1.0/docs/concepts/environments.md +55 -0
- brickie_core-0.1.0/docs/concepts/expectations.md +49 -0
- brickie_core-0.1.0/docs/concepts/how-brickie-reads-a-gold-models-sql.md +66 -0
- brickie_core-0.1.0/docs/concepts/naming.md +77 -0
- brickie_core-0.1.0/docs/concepts/the-databricks-cli.md +48 -0
- brickie_core-0.1.0/docs/figures/.gitkeep +0 -0
- brickie_core-0.1.0/docs/get-started/.gitkeep +0 -0
- brickie_core-0.1.0/docs/guides/.gitkeep +0 -0
- brickie_core-0.1.0/docs/guides/recover-from-a-failed-refresh.md +51 -0
- brickie_core-0.1.0/docs/guides/schedule-refreshes.md +73 -0
- brickie_core-0.1.0/docs/index.md +13 -0
- brickie_core-0.1.0/docs/reference/commands/compile.md +43 -0
- brickie_core-0.1.0/docs/reference/commands/databricks.md +14 -0
- brickie_core-0.1.0/docs/reference/commands/deploy.md +34 -0
- brickie_core-0.1.0/docs/reference/commands/init.md +51 -0
- brickie_core-0.1.0/docs/reference/commands/schema.md +40 -0
- brickie_core-0.1.0/docs/reference/commands/skills.md +38 -0
- brickie_core-0.1.0/docs/reference/commands/validate.md +33 -0
- brickie_core-0.1.0/docs/reference/json-output.md +87 -0
- brickie_core-0.1.0/docs/reference/kinds/model.md +16 -0
- brickie_core-0.1.0/docs/reference/kinds/platform.md +29 -0
- brickie_core-0.1.0/docs/reference/kinds/seed.md +18 -0
- brickie_core-0.1.0/docs/reference/kinds/source.md +27 -0
- brickie_core-0.1.0/docs/reference/rules/brk101.md +30 -0
- brickie_core-0.1.0/docs/reference/rules/brk102.md +31 -0
- brickie_core-0.1.0/docs/reference/rules/brk103.md +30 -0
- brickie_core-0.1.0/docs/reference/rules/brk104.md +31 -0
- brickie_core-0.1.0/docs/reference/rules/brk105.md +31 -0
- brickie_core-0.1.0/docs/reference/rules/brk106.md +31 -0
- brickie_core-0.1.0/docs/reference/rules/brk107.md +31 -0
- brickie_core-0.1.0/docs/reference/rules/brk108.md +29 -0
- brickie_core-0.1.0/docs/reference/rules/brk109.md +30 -0
- brickie_core-0.1.0/docs/reference/rules/brk110.md +37 -0
- brickie_core-0.1.0/docs/reference/rules/brk111.md +37 -0
- brickie_core-0.1.0/docs/reference/rules/brk112.md +32 -0
- brickie_core-0.1.0/docs/reference/rules/brk113.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk114.md +40 -0
- brickie_core-0.1.0/docs/reference/rules/brk115.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk116.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk117.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk118.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk119.md +38 -0
- brickie_core-0.1.0/docs/reference/rules/brk120.md +31 -0
- brickie_core-0.1.0/docs/reference/rules/brk121.md +31 -0
- brickie_core-0.1.0/docs/reference/rules/brk122.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk123.md +43 -0
- brickie_core-0.1.0/docs/reference/rules/brk124.md +41 -0
- brickie_core-0.1.0/docs/reference/rules/brk125.md +41 -0
- brickie_core-0.1.0/docs/reference/rules/brk126.md +40 -0
- brickie_core-0.1.0/docs/reference/rules/brk127.md +37 -0
- brickie_core-0.1.0/docs/reference/rules/brk129.md +35 -0
- brickie_core-0.1.0/docs/reference/rules/brk130.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk131.md +44 -0
- brickie_core-0.1.0/docs/reference/rules/brk132.md +41 -0
- brickie_core-0.1.0/docs/reference/rules/brk201.md +33 -0
- brickie_core-0.1.0/docs/reference/rules/brk202.md +39 -0
- brickie_core-0.1.0/docs/reference/rules/brk203.md +30 -0
- brickie_core-0.1.0/docs/reference/rules/brk204.md +33 -0
- brickie_core-0.1.0/docs/reference/rules/brk205.md +33 -0
- brickie_core-0.1.0/docs/reference/rules/brk206.md +33 -0
- brickie_core-0.1.0/docs/reference/rules/brk207.md +29 -0
- brickie_core-0.1.0/docs/reference/rules/brk208.md +29 -0
- brickie_core-0.1.0/docs/reference/rules/brk209.md +61 -0
- brickie_core-0.1.0/docs/reference/rules/brk210.md +33 -0
- brickie_core-0.1.0/docs/reference/rules/brk211.md +41 -0
- brickie_core-0.1.0/docs/reference/rules/brk212.md +46 -0
- brickie_core-0.1.0/docs/reference/rules/brk301.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk302.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk901.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk902.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk903.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk904.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk905.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk906.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk907.md +30 -0
- brickie_core-0.1.0/docs/reference/rules/brk908.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk909.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk910.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk911.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk912.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk913.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk914.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk915.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/brk916.md +26 -0
- brickie_core-0.1.0/docs/reference/rules/index.md +89 -0
- brickie_core-0.1.0/just/dev.just +4 -0
- brickie_core-0.1.0/just/docs.just +3 -0
- brickie_core-0.1.0/just/release.just +29 -0
- brickie_core-0.1.0/justfile +19 -0
- brickie_core-0.1.0/pyproject.toml +46 -0
- brickie_core-0.1.0/scripts/pin_databricks_cli.py +80 -0
- brickie_core-0.1.0/scripts/release.py +85 -0
- brickie_core-0.1.0/src/brickie/__init__.py +0 -0
- brickie_core-0.1.0/src/brickie/build/__init__.py +0 -0
- brickie_core-0.1.0/src/brickie/build/bundle.py +126 -0
- brickie_core-0.1.0/src/brickie/build/compiler.py +57 -0
- brickie_core-0.1.0/src/brickie/check/__init__.py +0 -0
- brickie_core-0.1.0/src/brickie/check/platform_objects.py +72 -0
- brickie_core-0.1.0/src/brickie/check/rules.py +371 -0
- brickie_core-0.1.0/src/brickie/cli.py +586 -0
- brickie_core-0.1.0/src/brickie/conditions.py +41 -0
- brickie_core-0.1.0/src/brickie/deploy/__init__.py +0 -0
- brickie_core-0.1.0/src/brickie/deploy/databricks.py +146 -0
- brickie_core-0.1.0/src/brickie/deploy/databricks_cli_release.py +13 -0
- brickie_core-0.1.0/src/brickie/deploy/deploy.py +71 -0
- brickie_core-0.1.0/src/brickie/deploy/pinned_cli.py +185 -0
- brickie_core-0.1.0/src/brickie/diagnostics.py +75 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/brickie.platform.yaml +21 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/data/day-1/customers/customers-day-1.json +40 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/data/day-1/orders/orders-day-1.json +150 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/data/day-1/product_prices/product_prices-day-1.json +6 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/data/day-2/customers/customers-day-2.json +2 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/data/day-2/orders/orders-day-2.json +100 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/data/day-2/product_prices/product_prices-day-2.json +1 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/finance/models/gold/fct_revenue.sql +10 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/finance/models/gold/fct_revenue.yaml +4 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/models/gold/dim_product_prices.sql +10 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/models/gold/dim_product_prices.yaml +4 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/models/gold/fct_orders.sql +16 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/models/gold/fct_orders.yaml +4 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/seeds/regions.csv +5 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/seeds/regions.yaml +7 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/sources/customers.yaml +14 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/sources/orders.yaml +10 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/sales/sources/product_prices.yaml +5 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/shared/models/gold/dim_customers.sql +9 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/shared/models/gold/dim_customers.yaml +4 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/shared/models/gold/dim_date.sql +9 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/domains/shared/models/gold/dim_date.yaml +4 -0
- brickie_core-0.1.0/src/brickie/examples/kettle/gitignore +1 -0
- brickie_core-0.1.0/src/brickie/kinds/__init__.py +24 -0
- brickie_core-0.1.0/src/brickie/kinds/base.py +75 -0
- brickie_core-0.1.0/src/brickie/kinds/model.py +109 -0
- brickie_core-0.1.0/src/brickie/kinds/platform.py +443 -0
- brickie_core-0.1.0/src/brickie/kinds/seed.py +279 -0
- brickie_core-0.1.0/src/brickie/kinds/source.py +346 -0
- brickie_core-0.1.0/src/brickie/load/__init__.py +0 -0
- brickie_core-0.1.0/src/brickie/load/config.py +155 -0
- brickie_core-0.1.0/src/brickie/load/project.py +203 -0
- brickie_core-0.1.0/src/brickie/names.py +153 -0
- brickie_core-0.1.0/src/brickie/output.py +49 -0
- brickie_core-0.1.0/src/brickie/reference.py +351 -0
- brickie_core-0.1.0/src/brickie/registry.py +967 -0
- brickie_core-0.1.0/src/brickie/scaffold.py +116 -0
- brickie_core-0.1.0/src/brickie/skills.py +185 -0
- brickie_core-0.1.0/src/brickie/sql.py +261 -0
- brickie_core-0.1.0/src/brickie/statements.py +25 -0
- brickie_core-0.1.0/src/brickie/templates/ci/github/.github/workflows/brickie-merge.yaml +24 -0
- brickie_core-0.1.0/src/brickie/templates/ci/github/.github/workflows/brickie-pull-request.yaml +23 -0
- brickie_core-0.1.0/src/brickie/templates/project/brickie.platform.yaml +17 -0
- brickie_core-0.1.0/src/brickie/templates/project/domains/sales/models/gold/dim_date.sql +9 -0
- brickie_core-0.1.0/src/brickie/templates/project/domains/sales/models/gold/dim_date.yaml +4 -0
- brickie_core-0.1.0/src/brickie/templates/project/domains/sales/models/gold/fct_orders.sql +2 -0
- brickie_core-0.1.0/src/brickie/templates/project/domains/sales/models/gold/fct_orders.yaml +4 -0
- brickie_core-0.1.0/src/brickie/templates/project/domains/sales/sources/orders.yaml +5 -0
- brickie_core-0.1.0/src/brickie/templates/project/gitignore +1 -0
- brickie_core-0.1.0/src/brickie/templates/skills/brickie-add-model/SKILL.md +17 -0
- brickie_core-0.1.0/src/brickie/templates/skills/brickie-add-source/SKILL.md +16 -0
- brickie_core-0.1.0/tests/conftest.py +38 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/dev.sql +95 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/databricks.yaml +173 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/finance/ingest/cost_centres.sql +8 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/sales/gold/fct_orders.sql +6 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/sales/gold/new_customers.sql +6 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/sales/gold/revenue.sql +4 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/sales/ingest/countries.sql +12 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/sales/ingest/customers.sql +8 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/sales/ingest/orders.sql +31 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/package/src/sales/ingest/refunds.sql +12 -0
- brickie_core-0.1.0/tests/fixtures/v1/expected/prod.sql +95 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/brickie.platform.yaml +17 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/finance/seeds/cost_centres.csv +1 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/finance/seeds/cost_centres.yaml +6 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/models/gold/fct_orders.sql +5 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/models/gold/fct_orders.yaml +3 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/models/gold/new_customers.sql +5 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/models/gold/new_customers.yaml +3 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/models/gold/revenue.yaml +3 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/models/gold/revenue_query.sql +3 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/seeds/countries.csv +4 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/seeds/countries.yaml +9 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/sources/customers.yaml +4 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/sources/orders.yaml +21 -0
- brickie_core-0.1.0/tests/fixtures/v1/project/domains/sales/sources/refunds.yaml +7 -0
- brickie_core-0.1.0/tests/test_build_order.py +256 -0
- brickie_core-0.1.0/tests/test_command_name.py +31 -0
- brickie_core-0.1.0/tests/test_dependencies_from_sql.py +513 -0
- brickie_core-0.1.0/tests/test_docs.py +213 -0
- brickie_core-0.1.0/tests/test_environment_roles.py +153 -0
- brickie_core-0.1.0/tests/test_expectations.py +328 -0
- brickie_core-0.1.0/tests/test_identifier_names.py +121 -0
- brickie_core-0.1.0/tests/test_init_folder.py +96 -0
- brickie_core-0.1.0/tests/test_json_output.py +522 -0
- brickie_core-0.1.0/tests/test_kettle.py +168 -0
- brickie_core-0.1.0/tests/test_kind_models.py +153 -0
- brickie_core-0.1.0/tests/test_kinds.py +258 -0
- brickie_core-0.1.0/tests/test_layers.py +246 -0
- brickie_core-0.1.0/tests/test_model_cycles.py +88 -0
- brickie_core-0.1.0/tests/test_naming_style_by_name.py +217 -0
- brickie_core-0.1.0/tests/test_naming_styles.py +173 -0
- brickie_core-0.1.0/tests/test_pin_databricks_cli.py +66 -0
- brickie_core-0.1.0/tests/test_pinned_cli.py +333 -0
- brickie_core-0.1.0/tests/test_pinned_output.py +47 -0
- brickie_core-0.1.0/tests/test_placeholders_in_model_sql.py +142 -0
- brickie_core-0.1.0/tests/test_platform_checks.py +206 -0
- brickie_core-0.1.0/tests/test_platform_contract.py +283 -0
- brickie_core-0.1.0/tests/test_reference.py +340 -0
- brickie_core-0.1.0/tests/test_refresh_failures.py +288 -0
- brickie_core-0.1.0/tests/test_refresh_job.py +207 -0
- brickie_core-0.1.0/tests/test_release.py +166 -0
- brickie_core-0.1.0/tests/test_seeds.py +333 -0
- brickie_core-0.1.0/tests/test_silver_write_patterns.py +268 -0
- brickie_core-0.1.0/tests/test_skills.py +293 -0
- brickie_core-0.1.0/tests/test_source_path.py +175 -0
- brickie_core-0.1.0/tests/test_statements.py +19 -0
- brickie_core-0.1.0/tests/test_streaming_gold_models.py +123 -0
- brickie_core-0.1.0/tests/test_unread_files.py +99 -0
- brickie_core-0.1.0/tests/test_walking_skeleton.py +228 -0
- brickie_core-0.1.0/tests/test_yaml_files.py +33 -0
- brickie_core-0.1.0/uv.lock +772 -0
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
pull_request:
|
|
5
|
+
push:
|
|
6
|
+
branches: [main]
|
|
7
|
+
workflow_call:
|
|
8
|
+
|
|
9
|
+
permissions:
|
|
10
|
+
contents: read
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
check:
|
|
14
|
+
runs-on: ubuntu-latest
|
|
15
|
+
steps:
|
|
16
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
17
|
+
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
|
|
18
|
+
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0
|
|
19
|
+
- run: just check
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
name: Release
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
tags: ["v*"]
|
|
6
|
+
|
|
7
|
+
permissions:
|
|
8
|
+
contents: read
|
|
9
|
+
|
|
10
|
+
jobs:
|
|
11
|
+
check:
|
|
12
|
+
uses: ./.github/workflows/ci.yaml
|
|
13
|
+
|
|
14
|
+
publish:
|
|
15
|
+
needs: check
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
environment: pypi
|
|
18
|
+
permissions:
|
|
19
|
+
contents: write # create the GitHub Release
|
|
20
|
+
id-token: write # Trusted Publishing to PyPI
|
|
21
|
+
steps:
|
|
22
|
+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
|
23
|
+
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
|
|
24
|
+
- uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4.0.0
|
|
25
|
+
- run: just release::build "$GITHUB_REF_NAME"
|
|
26
|
+
- run: just release::publish
|
|
27
|
+
- run: just release::github "$GITHUB_REF_NAME"
|
|
28
|
+
env:
|
|
29
|
+
GH_TOKEN: ${{ github.token }}
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
repos:
|
|
2
|
+
- repo: local
|
|
3
|
+
hooks:
|
|
4
|
+
- id: black
|
|
5
|
+
name: black
|
|
6
|
+
entry: uv run black --check src tests scripts
|
|
7
|
+
language: system
|
|
8
|
+
types: [python]
|
|
9
|
+
pass_filenames: false
|
|
10
|
+
- id: ruff
|
|
11
|
+
name: ruff
|
|
12
|
+
entry: uv run ruff check src tests scripts
|
|
13
|
+
language: system
|
|
14
|
+
types: [python]
|
|
15
|
+
pass_filenames: false
|
|
16
|
+
- id: ty
|
|
17
|
+
name: ty
|
|
18
|
+
entry: uv run ty check src tests scripts
|
|
19
|
+
language: system
|
|
20
|
+
types: [python]
|
|
21
|
+
pass_filenames: false
|
|
22
|
+
- id: pytest
|
|
23
|
+
name: pytest
|
|
24
|
+
entry: uv run pytest
|
|
25
|
+
language: system
|
|
26
|
+
always_run: true
|
|
27
|
+
pass_filenames: false
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
3.12
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Working agreement
|
|
2
|
+
|
|
3
|
+
Read before any work in this repo. These rules are non-negotiable.
|
|
4
|
+
|
|
5
|
+
## Building issues
|
|
6
|
+
|
|
7
|
+
- Build every issue test-first with `/tdd` (red → green → refactor) unless it has no testable behaviour (pure docs, config-only changes). Say so in the PR when skipping.
|
|
8
|
+
- Work in tracer-bullet vertical slices: each issue lands an end-to-end path, not a layer.
|
|
9
|
+
|
|
10
|
+
## Writing docs
|
|
11
|
+
|
|
12
|
+
Every PR that changes what a user sees or does updates its docs pages in the same PR, and adds its line to `## Unreleased` in `docs/changelog.md`. A change that needs an edit to a user's project says what to change.
|
|
13
|
+
|
|
14
|
+
### One job per page
|
|
15
|
+
|
|
16
|
+
- **Get started** takes you down one path to a working result.
|
|
17
|
+
- **A guide** does one task, and its title is that task: "Add a source".
|
|
18
|
+
- **A concept page** explains how Brickie thinks about one idea, and why.
|
|
19
|
+
- **Reference** is generated. Don't write it by hand.
|
|
20
|
+
|
|
21
|
+
A page never mixes these. If a guide starts explaining an idea, link to the concept page.
|
|
22
|
+
|
|
23
|
+
### Voice
|
|
24
|
+
|
|
25
|
+
- Write to "you", in the present tense, in short sentences.
|
|
26
|
+
- Say what happens. Not "Brickie allows you to quarantine rows" but "rows that fail go to quarantine".
|
|
27
|
+
- No marketing words: no "powerful", "seamless", "simply", "easily".
|
|
28
|
+
- Brickie's terms are lowercase in prose: a source, a gold model, quarantine, the platform contract. Identifiers stay in code: `keys`, `brickie.platform.yaml`, `BRK209`.
|
|
29
|
+
|
|
30
|
+
### Show, then tell
|
|
31
|
+
|
|
32
|
+
- Each guide step is a command or a file, followed by the real output it produces.
|
|
33
|
+
- Write commands bare, `brickie validate`, for a reader whose project's `.venv` is active.
|
|
34
|
+
- Examples come from Kettle, the fictional tea shop in `src/brickie/examples/kettle/`. A code block titled with a file path, ```` ```yaml title="domains/sales/sources/orders.yaml" ````, must match that file in Kettle. Reference pages use placeholders such as `<source>`.
|
|
35
|
+
|
|
36
|
+
### Link, don't repeat
|
|
37
|
+
|
|
38
|
+
- Each idea is explained on one page. Every other page links to it.
|
|
39
|
+
- A guide ends with "Next steps".
|
|
40
|
+
|
|
41
|
+
### Notes
|
|
42
|
+
|
|
43
|
+
- Use notes rarely. `> **Note**` is for what would surprise a careful reader. `> **Warning**` is only for where data could be lost.
|
|
44
|
+
- Use plain Markdown, so pages read the same on GitHub.
|
|
45
|
+
|
|
46
|
+
## Design
|
|
47
|
+
|
|
48
|
+
- Follow SOLID: small modules with one reason to change, depend on interfaces rather than concrete implementations where a seam is genuinely needed, and extend by adding code rather than editing stable code.
|
|
49
|
+
- Readability beats abstraction. Code is read far more than it is written, so prefer the plain, obvious version. Don't add an interface, base class, factory or layer of indirection until there is a second concrete use or a test that needs the seam. Three similar lines are better than a premature abstraction.
|
|
50
|
+
- A reader should be able to follow a module top to bottom without jumping through several files.
|
|
51
|
+
|
|
52
|
+
## Python toolchain
|
|
53
|
+
|
|
54
|
+
| Concern | Tool |
|
|
55
|
+
| --- | --- |
|
|
56
|
+
| Environments, dependencies, running | `uv` (`uv sync`, `uv run`, `uv add`); no pip, no manual venvs |
|
|
57
|
+
| Formatting | `black` |
|
|
58
|
+
| Linting | `ruff` (lint only; `black` owns formatting, so `ruff format` is not used) |
|
|
59
|
+
| Type checking | `ty` |
|
|
60
|
+
| Tests | `pytest`, run via `uv run` |
|
|
61
|
+
| Gate | `pre-commit` runs all of the above; CI runs the same hooks |
|
|
62
|
+
|
|
63
|
+
Nothing merges with a failing hook. Don't add `# noqa`, `# type: ignore` or skipped tests without a comment explaining why.
|
|
64
|
+
|
|
65
|
+
Any additional code-quality tooling (coverage thresholds, dependency license checks, secret scanning, dependency auditing) is welcome and, once added, equally non-negotiable.
|
|
66
|
+
|
|
67
|
+
## Task runner
|
|
68
|
+
|
|
69
|
+
- `just` is the single entry point for every routine task; CI calls the same recipes.
|
|
70
|
+
- Keep the root `justfile` lean: it holds only top-level recipes (`check`, `test`, `fmt`, `lint`, `typecheck`) and imports modules. Group the rest in modules under `just/` (e.g. `just/dev.just`, `just/release.just`, `just/databricks.just`) via `mod`, so each file stays short and readable.
|
|
71
|
+
- A recipe that grows beyond a few lines becomes a script under `scripts/`, called from the recipe.
|
|
72
|
+
|
|
73
|
+
## CI/CD
|
|
74
|
+
|
|
75
|
+
- CI is a thin wrapper over `just`: workflow steps call recipes rather than inlining shell, so local runs and CI behave the same.
|
|
76
|
+
- Where steps repeat across workflows, extract them into reusable components (reusable workflows via `workflow_call`, composite actions under `.github/actions/`) rather than copying them. Do this when there is real repetition, not speculatively; one workflow that reads top to bottom beats a maze of tiny pieces.
|
|
77
|
+
- Pin third-party actions to a version or commit SHA.
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
@AGENTS.md
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
|
|
2
|
+
Apache License
|
|
3
|
+
Version 2.0, January 2004
|
|
4
|
+
http://www.apache.org/licenses/
|
|
5
|
+
|
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
7
|
+
|
|
8
|
+
1. Definitions.
|
|
9
|
+
|
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
12
|
+
|
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
14
|
+
the copyright owner that is granting the License.
|
|
15
|
+
|
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
17
|
+
other entities that control, are controlled by, or are under common
|
|
18
|
+
control with that entity. For the purposes of this definition,
|
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
20
|
+
direction or management of such entity, whether by contract or
|
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
23
|
+
|
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
25
|
+
exercising permissions granted by this License.
|
|
26
|
+
|
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
28
|
+
including but not limited to software source code, documentation
|
|
29
|
+
source, and configuration files.
|
|
30
|
+
|
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
|
32
|
+
transformation or translation of a Source form, including but
|
|
33
|
+
not limited to compiled object code, generated documentation,
|
|
34
|
+
and conversions to other media types.
|
|
35
|
+
|
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
37
|
+
Object form, made available under the License, as indicated by a
|
|
38
|
+
copyright notice that is included in or attached to the work
|
|
39
|
+
(an example is provided in the Appendix below).
|
|
40
|
+
|
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
47
|
+
the Work and Derivative Works thereof.
|
|
48
|
+
|
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
|
50
|
+
the original version of the Work and any modifications or additions
|
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
62
|
+
|
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
65
|
+
subsequently incorporated within the Work.
|
|
66
|
+
|
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
|
73
|
+
|
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
79
|
+
where such license applies only to those patent claims licensable
|
|
80
|
+
by such Contributor that are necessarily infringed by their
|
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
83
|
+
institute patent litigation against any entity (including a
|
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
86
|
+
or contributory patent infringement, then any patent licenses
|
|
87
|
+
granted to You under this License for that Work shall terminate
|
|
88
|
+
as of the date such litigation is filed.
|
|
89
|
+
|
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
92
|
+
modifications, and in Source or Object form, provided that You
|
|
93
|
+
meet the following conditions:
|
|
94
|
+
|
|
95
|
+
(a) You must give any other recipients of the Work or
|
|
96
|
+
Derivative Works a copy of this License; and
|
|
97
|
+
|
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
|
99
|
+
stating that You changed the files; and
|
|
100
|
+
|
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
|
103
|
+
attribution notices from the Source form of the Work,
|
|
104
|
+
excluding those notices that do not pertain to any part of
|
|
105
|
+
the Derivative Works; and
|
|
106
|
+
|
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
|
109
|
+
include a readable copy of the attribution notices contained
|
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
|
112
|
+
of the following places: within a NOTICE text file distributed
|
|
113
|
+
as part of the Derivative Works; within the Source form or
|
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
|
115
|
+
within a display generated by the Derivative Works, if and
|
|
116
|
+
wherever such third-party notices normally appear. The contents
|
|
117
|
+
of the NOTICE file are for informational purposes only and
|
|
118
|
+
do not modify the License. You may add Your own attribution
|
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
121
|
+
that such additional attribution notices cannot be construed
|
|
122
|
+
as modifying the License.
|
|
123
|
+
|
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
|
125
|
+
may provide additional or different license terms and conditions
|
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
129
|
+
the conditions stated in this License.
|
|
130
|
+
|
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
134
|
+
this License, without any additional terms or conditions.
|
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
136
|
+
the terms of any separate license agreement you may have executed
|
|
137
|
+
with Licensor regarding such Contributions.
|
|
138
|
+
|
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
141
|
+
except as required for reasonable and customary use in describing the
|
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
143
|
+
|
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
|
153
|
+
|
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
|
159
|
+
incidental, or consequential damages of any character arising as a
|
|
160
|
+
result of this License or out of the use or inability to use the
|
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
163
|
+
other commercial damages or losses), even if such Contributor
|
|
164
|
+
has been advised of the possibility of such damages.
|
|
165
|
+
|
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
169
|
+
or other liability obligations and/or rights consistent with this
|
|
170
|
+
License. However, in accepting such obligations, You may act only
|
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
175
|
+
of your accepting any such warranty or additional liability.
|
|
176
|
+
|
|
177
|
+
END OF TERMS AND CONDITIONS
|
|
178
|
+
|
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
180
|
+
|
|
181
|
+
To apply the Apache License to your work, attach the following
|
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
183
|
+
replaced with your own identifying information. (Don't include
|
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
185
|
+
comment syntax for the file format. We also recommend that a
|
|
186
|
+
file or class name and description of purpose be included on the
|
|
187
|
+
same "printed page" as the copyright notice for easier
|
|
188
|
+
identification within third-party archives.
|
|
189
|
+
|
|
190
|
+
Copyright [yyyy] [name of copyright owner]
|
|
191
|
+
|
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
193
|
+
you may not use this file except in compliance with the License.
|
|
194
|
+
You may obtain a copy of the License at
|
|
195
|
+
|
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
197
|
+
|
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
201
|
+
See the License for the specific language governing permissions and
|
|
202
|
+
limitations under the License.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: brickie-core
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Bring the software development lifecycle to Databricks pipelines
|
|
5
|
+
License-Expression: Apache-2.0
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
Requires-Python: >=3.12
|
|
8
|
+
Requires-Dist: click>=8.3
|
|
9
|
+
Requires-Dist: pydantic>=2.14.2
|
|
10
|
+
Requires-Dist: pyyaml>=6.0.3
|
|
11
|
+
Requires-Dist: sqlglot==26.*
|
|
12
|
+
Requires-Dist: typer>=0.27.3
|
|
13
|
+
Description-Content-Type: text/markdown
|
|
14
|
+
|
|
15
|
+
<h1 align=center><code>brickie</code></h1>
|
|
16
|
+
|
|
17
|
+
<div align=center>
|
|
18
|
+
<a href=https://github.com/nimblestax/brickie-core/actions/workflows/ci.yaml>
|
|
19
|
+
<img src=https://github.com/nimblestax/brickie-core/actions/workflows/ci.yaml/badge.svg alt="build status">
|
|
20
|
+
</a>
|
|
21
|
+
<a href=LICENSE>
|
|
22
|
+
<img src=https://img.shields.io/badge/license-Apache%202.0-blue.svg alt="Apache 2.0 license">
|
|
23
|
+
</a>
|
|
24
|
+
</div>
|
|
25
|
+
<br>
|
|
26
|
+
|
|
27
|
+
Brickie brings the software development lifecycle to Databricks. Today it covers pipelines, from bronze to gold: config-driven, reviewed in pull requests, tested, and promoted from dev to prod. It grows to cover more of the platform.
|
|
28
|
+
|
|
29
|
+
**Status:** early development, and below 1.0, so a minor release may need edits to your project. Install it into your project with `uv add --dev brickie-core`. The Plan shown on [nimblestax.com](https://nimblestax.com) is illustrative: Brickie has no Plan command today.
|
|
30
|
+
|
|
31
|
+
A pipeline is YAML and SQL in your repo. A Source says where data lands:
|
|
32
|
+
|
|
33
|
+
```yaml
|
|
34
|
+
# domains/sales/sources/orders.yaml
|
|
35
|
+
kind: source
|
|
36
|
+
version: 1
|
|
37
|
+
path: /Volumes/{env}_sales/bronze/landing/orders
|
|
38
|
+
format: json
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
A Model is the SQL you'd write anyway, using Logical Names:
|
|
42
|
+
|
|
43
|
+
```sql
|
|
44
|
+
-- domains/sales/models/fct_orders.sql
|
|
45
|
+
SELECT order_id, amount
|
|
46
|
+
FROM sales.silver.orders
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Brickie compiles that for each Environment:
|
|
50
|
+
|
|
51
|
+
```console
|
|
52
|
+
$ brickie compile --env dev
|
|
53
|
+
CREATE OR REFRESH STREAMING TABLE dev_sales.bronze.orders
|
|
54
|
+
TBLPROPERTIES ('pipelines.reset.allowed' = 'false')
|
|
55
|
+
AS SELECT *
|
|
56
|
+
FROM STREAM read_files('/Volumes/dev_sales/bronze/landing/orders', format => 'json');
|
|
57
|
+
...
|
|
58
|
+
CREATE OR REFRESH MATERIALIZED VIEW dev_sales.gold.fct_orders AS
|
|
59
|
+
SELECT order_id, amount
|
|
60
|
+
FROM dev_sales.silver.orders
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
And it catches mistakes before anything touches a workspace:
|
|
64
|
+
|
|
65
|
+
```console
|
|
66
|
+
$ brickie validate
|
|
67
|
+
BRK201 domains/sales/models/fct_orders.sql: sales.silver.ordrs isn't a table in this project.
|
|
68
|
+
Fix: Declare a Source or Gold Model with this name, or correct the Logical Name.
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## What works today
|
|
72
|
+
|
|
73
|
+
- `brickie init` creates a working project, with GitHub Actions workflows.
|
|
74
|
+
- `brickie validate` checks config and SQL references with no workspace. A Model's dependencies come from its SQL, with no list to keep: CTEs, aliases and `STREAM(...)` just work, and anything Brickie can't read, such as `IDENTIFIER(...)`, is an error rather than a guess.
|
|
75
|
+
- `brickie compile` writes generated code, or prints it for one Environment.
|
|
76
|
+
- `brickie deploy` deploys to an Environment and refreshes its tables.
|
|
77
|
+
- `brickie schema` prints any Kind's fields.
|
|
78
|
+
- Every command has [`--json` output](docs/reference/json-output.md), rule IDs and fixed exit codes.
|
|
79
|
+
|
|
80
|
+
## The Nimblestax Way
|
|
81
|
+
|
|
82
|
+
Brickie is built around an opinionated way of building on Databricks. Read the principles at [nimblestax.com/way](https://nimblestax.com/way).
|
|
83
|
+
|
|
84
|
+
## Follow progress
|
|
85
|
+
|
|
86
|
+
Watch this repo's releases on GitHub.
|
|
87
|
+
|
|
88
|
+
## License
|
|
89
|
+
|
|
90
|
+
[Apache License 2.0](LICENSE). Contributions are accepted under the [Developer Certificate of Origin](https://developercertificate.org/): sign off each commit with `git commit -s`.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
<h1 align=center><code>brickie</code></h1>
|
|
2
|
+
|
|
3
|
+
<div align=center>
|
|
4
|
+
<a href=https://github.com/nimblestax/brickie-core/actions/workflows/ci.yaml>
|
|
5
|
+
<img src=https://github.com/nimblestax/brickie-core/actions/workflows/ci.yaml/badge.svg alt="build status">
|
|
6
|
+
</a>
|
|
7
|
+
<a href=LICENSE>
|
|
8
|
+
<img src=https://img.shields.io/badge/license-Apache%202.0-blue.svg alt="Apache 2.0 license">
|
|
9
|
+
</a>
|
|
10
|
+
</div>
|
|
11
|
+
<br>
|
|
12
|
+
|
|
13
|
+
Brickie brings the software development lifecycle to Databricks. Today it covers pipelines, from bronze to gold: config-driven, reviewed in pull requests, tested, and promoted from dev to prod. It grows to cover more of the platform.
|
|
14
|
+
|
|
15
|
+
**Status:** early development, and below 1.0, so a minor release may need edits to your project. Install it into your project with `uv add --dev brickie-core`. The Plan shown on [nimblestax.com](https://nimblestax.com) is illustrative: Brickie has no Plan command today.
|
|
16
|
+
|
|
17
|
+
A pipeline is YAML and SQL in your repo. A Source says where data lands:
|
|
18
|
+
|
|
19
|
+
```yaml
|
|
20
|
+
# domains/sales/sources/orders.yaml
|
|
21
|
+
kind: source
|
|
22
|
+
version: 1
|
|
23
|
+
path: /Volumes/{env}_sales/bronze/landing/orders
|
|
24
|
+
format: json
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
A Model is the SQL you'd write anyway, using Logical Names:
|
|
28
|
+
|
|
29
|
+
```sql
|
|
30
|
+
-- domains/sales/models/fct_orders.sql
|
|
31
|
+
SELECT order_id, amount
|
|
32
|
+
FROM sales.silver.orders
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Brickie compiles that for each Environment:
|
|
36
|
+
|
|
37
|
+
```console
|
|
38
|
+
$ brickie compile --env dev
|
|
39
|
+
CREATE OR REFRESH STREAMING TABLE dev_sales.bronze.orders
|
|
40
|
+
TBLPROPERTIES ('pipelines.reset.allowed' = 'false')
|
|
41
|
+
AS SELECT *
|
|
42
|
+
FROM STREAM read_files('/Volumes/dev_sales/bronze/landing/orders', format => 'json');
|
|
43
|
+
...
|
|
44
|
+
CREATE OR REFRESH MATERIALIZED VIEW dev_sales.gold.fct_orders AS
|
|
45
|
+
SELECT order_id, amount
|
|
46
|
+
FROM dev_sales.silver.orders
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
And it catches mistakes before anything touches a workspace:
|
|
50
|
+
|
|
51
|
+
```console
|
|
52
|
+
$ brickie validate
|
|
53
|
+
BRK201 domains/sales/models/fct_orders.sql: sales.silver.ordrs isn't a table in this project.
|
|
54
|
+
Fix: Declare a Source or Gold Model with this name, or correct the Logical Name.
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## What works today
|
|
58
|
+
|
|
59
|
+
- `brickie init` creates a working project, with GitHub Actions workflows.
|
|
60
|
+
- `brickie validate` checks config and SQL references with no workspace. A Model's dependencies come from its SQL, with no list to keep: CTEs, aliases and `STREAM(...)` just work, and anything Brickie can't read, such as `IDENTIFIER(...)`, is an error rather than a guess.
|
|
61
|
+
- `brickie compile` writes generated code, or prints it for one Environment.
|
|
62
|
+
- `brickie deploy` deploys to an Environment and refreshes its tables.
|
|
63
|
+
- `brickie schema` prints any Kind's fields.
|
|
64
|
+
- Every command has [`--json` output](docs/reference/json-output.md), rule IDs and fixed exit codes.
|
|
65
|
+
|
|
66
|
+
## The Nimblestax Way
|
|
67
|
+
|
|
68
|
+
Brickie is built around an opinionated way of building on Databricks. Read the principles at [nimblestax.com/way](https://nimblestax.com/way).
|
|
69
|
+
|
|
70
|
+
## Follow progress
|
|
71
|
+
|
|
72
|
+
Watch this repo's releases on GitHub.
|
|
73
|
+
|
|
74
|
+
## License
|
|
75
|
+
|
|
76
|
+
[Apache License 2.0](LICENSE). Contributions are accepted under the [Developer Certificate of Origin](https://developercertificate.org/): sign off each commit with `git commit -s`.
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Changelog
|
|
3
|
+
description: What changed in each release of Brickie, newest first.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Every release of Brickie, newest first. Each section is also that release's notes on GitHub.
|
|
7
|
+
|
|
8
|
+
Each pull request that changes what you see adds its line under Unreleased, so each release's section is built up by its pull requests. A release renames Unreleased to its version.
|
|
9
|
+
|
|
10
|
+
Brickie follows [Semantic Versioning](https://semver.org). Before 1.0, a minor release, such as 0.2.0, may change config or commands in ways that need edits to your project, and its section here says what to change. A patch release, such as 0.2.1, never does.
|
|
11
|
+
|
|
12
|
+
## Unreleased
|
|
13
|
+
|
|
14
|
+
## 0.1.0
|
|
15
|
+
|
|
16
|
+
The first release. Install it into your project with `uv add --dev brickie-core`.
|
|
17
|
+
|
|
18
|
+
- [`brickie init`](reference/commands/init.md) creates a project folder and asks where your tables live: `domain-catalogs` (`dev_sales.silver.orders`, the default), `layer-catalogs` (`dev_silver.sales.orders`) or `environment-catalogs` (`dev.sales_silver.orders`). `brickie.platform.yaml` keeps that choice as `naming.style`. See [Naming](concepts/naming.md).
|
|
19
|
+
- It refuses a folder that already holds `brickie.platform.yaml`, with [`BRK909`](reference/rules/brk909.md), and writes nothing.
|
|
20
|
+
- `--ci github` adds GitHub Actions workflows, `.github/workflows/brickie-pull-request.yaml` and `.github/workflows/brickie-merge.yaml`. They install `brickie-core` at the version `BRICKIE_SPEC` names.
|
|
21
|
+
- It lists the Volumes the project's sources read in dev, with the catalogs and schemas to create.
|
|
22
|
+
- `--example kettle` writes Kettle, the fictional tea shop every docs page draws its examples from, with two days of sample data and the commands that upload day 1.
|
|
23
|
+
- Environments are named freely and carry a role: `role: development` on exactly one, and `role: production` on at most one. Any other environment is a stage. Commands that read default to the development environment, and an environment without a `host` uses its host. See [Environments](concepts/environments.md).
|
|
24
|
+
- [`brickie validate`](reference/commands/validate.md) checks config and SQL with no workspace. A gold model's dependencies come from its SQL, and anything Brickie can't read, such as `IDENTIFIER(...)`, is an error rather than a guess.
|
|
25
|
+
- Tables are read in build order. A gold model reads silver, a gold model in its own domain, another domain's exported silver, or gold in the shared domain. See [Domains and layers](concepts/domains-and-layers.md).
|
|
26
|
+
- Model layers have no order, and a gold model's folder sets its layer. A loop of reads across model layers fails.
|
|
27
|
+
- `--platform` signs in to an environment's workspace and checks that the catalogs and schemas the project needs are there.
|
|
28
|
+
- It warns when a skill was written by another Brickie version.
|
|
29
|
+
- [`brickie compile`](reference/commands/compile.md) writes the generated code, or prints it for one environment. A gold model that reads `STREAM(...)` compiles to a streaming table, and any other gold model to a materialized view. See [How Brickie reads a gold model's SQL](concepts/how-brickie-reads-a-gold-models-sql.md).
|
|
30
|
+
- [`brickie deploy`](reference/commands/deploy.md) deploys to the environment `--env` names and refreshes its tables. When a refresh fails, it says why. See [Recover from a failed refresh](guides/recover-from-a-failed-refresh.md).
|
|
31
|
+
- Each environment has one Job, named for it, such as `kettle_dev_refresh`. It waits only where one pipeline reads another's tables, so a failed pipeline skips only what reads from it.
|
|
32
|
+
- An environment can refresh on a schedule. A schedule runs in the production environment and is paused elsewhere. See [Schedule refreshes](guides/schedule-refreshes.md).
|
|
33
|
+
- [`brickie schema`](reference/commands/schema.md) prints any Kind's fields.
|
|
34
|
+
- A source lands in bronze, and silver either appends it or upserts it by `keys` and `sequence_by`, with Type 1 or Type 2 history and `delete_when` for deletes.
|
|
35
|
+
- Seeds load small reference tables from CSV files in the repo.
|
|
36
|
+
- Expectations check silver rows, and rows that fail are kept in quarantine. See [Expectations](concepts/expectations.md).
|
|
37
|
+
- Brickie brings its own Databricks CLI, version 1.23.0, and never runs the `databricks` on your PATH. [`brickie databricks`](reference/commands/databricks.md) runs it, and `BRICKIE_DATABRICKS_CLI` points Brickie at one you installed yourself. See [The Databricks CLI](concepts/the-databricks-cli.md).
|
|
38
|
+
- `brickie init` writes Brickie skills for coding agents, `brickie-add-source` and `brickie-add-model`, into `.agents/skills/` and `.claude/skills/`. [`brickie skills update`](reference/commands/skills.md#update) refreshes them. See [Brickie skills](concepts/brickie-skills.md).
|
|
39
|
+
- Every command but `brickie databricks` has `--json` output, with Rule IDs and fixed exit codes. See [JSON output](reference/json-output.md).
|
|
40
|
+
- Every Rule has its own reference page, listed in the [Rules index](reference/rules/index.md).
|
|
File without changes
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Brickie skills
|
|
3
|
+
description: The agent skills brickie init writes into your repo, why there are two copies, and how they stay in step with Brickie.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
A Brickie skill guides a coding agent through one task with Brickie's commands, such as adding a source. `brickie init` writes them into your project, and you commit them like any other file.
|
|
7
|
+
|
|
8
|
+
```console
|
|
9
|
+
$ brickie init kettle
|
|
10
|
+
...
|
|
11
|
+
Wrote Brickie Skills for coding agents:
|
|
12
|
+
.agents/skills/brickie-add-model
|
|
13
|
+
.claude/skills/brickie-add-model
|
|
14
|
+
.agents/skills/brickie-add-source
|
|
15
|
+
.claude/skills/brickie-add-source
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Each skill is one `SKILL.md` that names the commands to run. It doesn't list a kind's fields. It tells the agent to run `brickie schema <kind>`, so the fields come from the Brickie you have installed. No skill deploys: an agent writes the change, and a person promotes it.
|
|
19
|
+
|
|
20
|
+
## Where they go
|
|
21
|
+
|
|
22
|
+
Skills follow the [Agent Skills](https://agentskills.io) standard and use only its fields, so every agent that reads the standard gets the same steps.
|
|
23
|
+
|
|
24
|
+
`.agents/skills/` holds the main copy, in the folder many coding agents share. Claude Code reads only `.claude/skills/`, so it gets an identical copy. They are copies, not symlinks, because symlinks don't check out reliably on Windows.
|
|
25
|
+
|
|
26
|
+
Brickie only ever adds `brickie-*` folders. Your own skills, settings and other files in `.agents/` and `.claude/` stay as they are, and a `brickie-*` folder that already exists is left alone. If Git ignores either folder, `brickie init` warns with [`BRK914`](../reference/rules/brk914.md), because skills that aren't committed stay on one machine.
|
|
27
|
+
|
|
28
|
+
To write no skills, run `brickie init --skills none`. `brickie skills update` writes them later if you change your mind.
|
|
29
|
+
|
|
30
|
+
## Keeping them in step
|
|
31
|
+
|
|
32
|
+
Each skill records the Brickie version that wrote it, in its front matter:
|
|
33
|
+
|
|
34
|
+
```yaml
|
|
35
|
+
metadata:
|
|
36
|
+
brickie-version: "0.1.0"
|
|
37
|
+
brickie-digest: "..."
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
After you upgrade Brickie, `brickie validate` warns with [`BRK129`](../reference/rules/brk129.md) for each skill an older version wrote, because its steps can name commands that have changed. A warning doesn't fail validation.
|
|
41
|
+
|
|
42
|
+
`brickie skills update` rewrites both copies of each skill to the installed version:
|
|
43
|
+
|
|
44
|
+
```console
|
|
45
|
+
$ brickie skills update
|
|
46
|
+
Updated these Brickie Skills. Commit them:
|
|
47
|
+
.agents/skills/brickie-add-source
|
|
48
|
+
.claude/skills/brickie-add-source
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
`brickie-digest` records the text Brickie wrote, so `brickie skills update` can tell your edit from an older Brickie's text. If you edited a skill, it writes nothing, reports [`BRK915`](../reference/rules/brk915.md) and shows the difference. Move your steps into a skill of your own, outside the `brickie-*` folders, or run `brickie skills update --force` to replace them.
|
|
52
|
+
|
|
53
|
+
## Next steps
|
|
54
|
+
|
|
55
|
+
- [`brickie skills update`](../reference/commands/skills.md#update) lists its options and its JSON result.
|
|
56
|
+
- [`brickie init`](../reference/commands/init.md) lists `--skills` with its other options.
|