dbt-bouncer 0.14.0__tar.gz

dbt_bouncer-0.14.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Pádraic Slattery
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dbt_bouncer-0.14.0/PKG-INFO

@@ -0,0 +1,196 @@
+Metadata-Version: 2.1
+Name: dbt-bouncer
+Version: 0.14.0
+Summary: Configure and enforce conventions for your dbt project.
+Home-page: https://github.com/godatadriven/dbt-bouncer
+License: MIT
+Keywords: python,cli,dbt,CI/CD
+Author: Padraic Slattery
+Author-email: pgoslatara@gmail.com
+Maintainer: Padraic Slattery
+Maintainer-email: pgoslatara@gmail.com
+Requires-Python: >=3.8.1,<3.13
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.8
+Requires-Dist: click (<9)
+Requires-Dist: dbt-artifacts-parser (>=0,<1)
+Requires-Dist: pydantic (>=2,<3)
+Requires-Dist: pytest (<9)
+Requires-Dist: pyyaml (<7)
+Requires-Dist: requests (>=2,<3)
+Requires-Dist: tabulate (>=0,<1)
+Project-URL: Repository, https://github.com/godatadriven/dbt-bouncer
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="./images/logo.webp" alt="dbt-bouncer logo" width="500"/>
+</p>
+
+
+<h1 align="center">
+  dbt-bouncer
+</h1>
+<h2 align="center">
+  Configure and enforce conventions for your dbt project.
+</h2>
+
+<div align="center">
+  <a>
+    <img src="https://img.shields.io/github/release/godatadriven/dbt-bouncer.svg?logo=github">
+  </a>
+  <a>
+    <img src="https://github.com/godatadriven/dbt-bouncer/actions/workflows/ci_pipeline.yml/badge.svg">
+  </a>
+  <a>
+    <img src="https://img.shields.io/badge/License-MIT-yellow.svg">
+  </a>
+  <a>
+    <img src="https://img.shields.io/github/last-commit/godatadriven/dbt-bouncer/main">
+  </a>
+  <a>
+    <img src="https://img.shields.io/github/commits-since/godatadriven/dbt-bouncer/latest">
+  </a>
+</div>
+
+<div align="center">
+  <a>
+    <img alt="dbt" src="https://img.shields.io/badge/dbt%20-%3E%3D1.6-333?logo=dbt">
+  </a>
+  <a>
+    <img alt="Docker Supported" src="https://img.shields.io/badge/Docker%20-Supported-0db7ed?logo=docker">
+  </a>
+  <a>
+    <img alt="GitHub Supported" src="https://img.shields.io/badge/GitHub%20-Supported-333?logo=github">
+  </a>
+</div>
+
+<div align="center">
+  <a>
+    <img src="https://img.shields.io/badge/code%20style-black-000000.svg">
+  </a>
+  <a>
+    <img src="https://www.aschey.tech/tokei/github/godatadriven/dbt-bouncer?category=code">
+  </a>
+</div>
+<br/>
+
+# How to use
+
+1. Generate a `manifest.json` by running `dbt parse`.
+1. Create a `dbt-bouncer.yml` config file, details [here](#config-file).
+1. Run `dbt-bouncer` to validate that your conventions are being maintained. You can use GitHub Actions, Docker, a `.pex` file or python to run `dbt-bouncer`.
+
+## GitHub Actions
+
+```yaml
+steps:
+    ...
+
+    - uses: godatadriven/dbt-bouncer@v0
+      with:
+        config-file: ./<PATH_TO_CONFIG_FILE>
+        send-pr-comment: true # optional, defaults to true
+
+    ...
+```
+
+## Docker
+
+Don't use GitHub Actions? You can still use dbt-bouncer via Docker:
+
+```bash
+docker run --rm \
+    --volume "$PWD":/app \
+    ghcr.io/godatadriven/dbt-bouncer:vX.X.X \
+    --config-file /app/<PATH_TO_CONFIG_FILE>
+```
+
+## Pex
+
+You can also run the `.pex` ([Python EXecutable](https://docs.pex-tool.org/whatispex.html#whatispex)) artifact directly once you have a python executable installed:
+
+```bash
+wget https://github.com/godatadriven/dbt-bouncer/releases/download/vX.X.X/dbt-bouncer.pex -O dbt-bouncer.pex
+
+dbt-bouncer.pex --config-file $PWD/<PATH_TO_CONFIG_FILE>
+```
+
+## Python
+
+Install from [pypi.org](https://pypi.org/dbt-bouncer):
+
+```shell
+pip install dbt-bouncer
+```
+
+Run:
+```shell
+dbt-bouncer.pex --config-file $PWD/<PATH_TO_CONFIG_FILE>
+``
+
+# Config file
+
+`dbt-bouncer` requires a config file to be provided. This file configures what checks are run. Here is an example config file:
+
+```yaml
+dbt-artifacts-dir: target # [Optional] Directory where the dbt artifacts exists, generally the `target` directory inside a dbt project. Defaults to `./target`.
+
+manifest_checks:
+  - name: check_macro_name_matches_file_name
+  - name: check_model_names
+    include: ^staging
+    model_name_pattern: ^stg_
+```
+
+# Checks
+
+:bulb: Click on a check name to see more details.
+
+**Macros**
+
+* [`check_macro_arguments_description_populated`](./dbt_bouncer/checks/checks.md#check_macro_arguments_description_populated): Macro arguments must have a populated description.
+* [`check_macro_description_populated`](./dbt_bouncer/checks/checks.md#check_macro_description_populated): Macros must have a populated description.
+* [`check_macro_name_matches_file_name`](./dbt_bouncer/checks/checks.md#check_macro_name_matches_file_name): Macros names must be the same as the file they are contained in.
+
+**Metadata**
+
+* [`check_project_name`](./dbt_bouncer/checks/checks.md#check_project_name): Enforce that the name of the dbt project matches a supplied regex.
+
+**Miscellaneous**
+
+* [`check_top_level_directories`](./dbt_bouncer/checks/checks.md#check_top_level_directories): Only specified top-level directories are allowed to contain models.
+
+**Models**
+
+* [`check_model_description_populated`](./dbt_bouncer/checks/checks.md#check_model_description_populated): Models must have a populated description.
+* [`check_model_names`](./dbt_bouncer/checks/checks.md#check_model_names): Models must have a name that matches the supplied regex.
+
+**Sources**
+
+* [`check_source_has_meta_keys`](./dbt_bouncer/checks/checks.md#check_source_has_meta_keys): The `meta` config for sources must have the specified keys.
+
+# Development
+
+To setup your development environment, fork this repository and run:
+
+```bash
+poetry shell
+poetry install
+pre-commit install
+```
+
+Set required environment variables by copying `.env.example` to `.env` and updating the values.
+
+All tests can be run via:
+```bash
+make build-artifacts # Rebuilds dbt artifacts used by pytest
+make test
+```
+

dbt_bouncer-0.14.0/README.md

@@ -0,0 +1,164 @@
+<p align="center">
+  <img src="./images/logo.webp" alt="dbt-bouncer logo" width="500"/>
+</p>
+
+
+<h1 align="center">
+  dbt-bouncer
+</h1>
+<h2 align="center">
+  Configure and enforce conventions for your dbt project.
+</h2>
+
+<div align="center">
+  <a>
+    <img src="https://img.shields.io/github/release/godatadriven/dbt-bouncer.svg?logo=github">
+  </a>
+  <a>
+    <img src="https://github.com/godatadriven/dbt-bouncer/actions/workflows/ci_pipeline.yml/badge.svg">
+  </a>
+  <a>
+    <img src="https://img.shields.io/badge/License-MIT-yellow.svg">
+  </a>
+  <a>
+    <img src="https://img.shields.io/github/last-commit/godatadriven/dbt-bouncer/main">
+  </a>
+  <a>
+    <img src="https://img.shields.io/github/commits-since/godatadriven/dbt-bouncer/latest">
+  </a>
+</div>
+
+<div align="center">
+  <a>
+    <img alt="dbt" src="https://img.shields.io/badge/dbt%20-%3E%3D1.6-333?logo=dbt">
+  </a>
+  <a>
+    <img alt="Docker Supported" src="https://img.shields.io/badge/Docker%20-Supported-0db7ed?logo=docker">
+  </a>
+  <a>
+    <img alt="GitHub Supported" src="https://img.shields.io/badge/GitHub%20-Supported-333?logo=github">
+  </a>
+</div>
+
+<div align="center">
+  <a>
+    <img src="https://img.shields.io/badge/code%20style-black-000000.svg">
+  </a>
+  <a>
+    <img src="https://www.aschey.tech/tokei/github/godatadriven/dbt-bouncer?category=code">
+  </a>
+</div>
+<br/>
+
+# How to use
+
+1. Generate a `manifest.json` by running `dbt parse`.
+1. Create a `dbt-bouncer.yml` config file, details [here](#config-file).
+1. Run `dbt-bouncer` to validate that your conventions are being maintained. You can use GitHub Actions, Docker, a `.pex` file or python to run `dbt-bouncer`.
+
+## GitHub Actions
+
+```yaml
+steps:
+    ...
+
+    - uses: godatadriven/dbt-bouncer@v0
+      with:
+        config-file: ./<PATH_TO_CONFIG_FILE>
+        send-pr-comment: true # optional, defaults to true
+
+    ...
+```
+
+## Docker
+
+Don't use GitHub Actions? You can still use dbt-bouncer via Docker:
+
+```bash
+docker run --rm \
+    --volume "$PWD":/app \
+    ghcr.io/godatadriven/dbt-bouncer:vX.X.X \
+    --config-file /app/<PATH_TO_CONFIG_FILE>
+```
+
+## Pex
+
+You can also run the `.pex` ([Python EXecutable](https://docs.pex-tool.org/whatispex.html#whatispex)) artifact directly once you have a python executable installed:
+
+```bash
+wget https://github.com/godatadriven/dbt-bouncer/releases/download/vX.X.X/dbt-bouncer.pex -O dbt-bouncer.pex
+
+dbt-bouncer.pex --config-file $PWD/<PATH_TO_CONFIG_FILE>
+```
+
+## Python
+
+Install from [pypi.org](https://pypi.org/dbt-bouncer):
+
+```shell
+pip install dbt-bouncer
+```
+
+Run:
+```shell
+dbt-bouncer.pex --config-file $PWD/<PATH_TO_CONFIG_FILE>
+``
+
+# Config file
+
+`dbt-bouncer` requires a config file to be provided. This file configures what checks are run. Here is an example config file:
+
+```yaml
+dbt-artifacts-dir: target # [Optional] Directory where the dbt artifacts exists, generally the `target` directory inside a dbt project. Defaults to `./target`.
+
+manifest_checks:
+  - name: check_macro_name_matches_file_name
+  - name: check_model_names
+    include: ^staging
+    model_name_pattern: ^stg_
+```
+
+# Checks
+
+:bulb: Click on a check name to see more details.
+
+**Macros**
+
+* [`check_macro_arguments_description_populated`](./dbt_bouncer/checks/checks.md#check_macro_arguments_description_populated): Macro arguments must have a populated description.
+* [`check_macro_description_populated`](./dbt_bouncer/checks/checks.md#check_macro_description_populated): Macros must have a populated description.
+* [`check_macro_name_matches_file_name`](./dbt_bouncer/checks/checks.md#check_macro_name_matches_file_name): Macros names must be the same as the file they are contained in.
+
+**Metadata**
+
+* [`check_project_name`](./dbt_bouncer/checks/checks.md#check_project_name): Enforce that the name of the dbt project matches a supplied regex.
+
+**Miscellaneous**
+
+* [`check_top_level_directories`](./dbt_bouncer/checks/checks.md#check_top_level_directories): Only specified top-level directories are allowed to contain models.
+
+**Models**
+
+* [`check_model_description_populated`](./dbt_bouncer/checks/checks.md#check_model_description_populated): Models must have a populated description.
+* [`check_model_names`](./dbt_bouncer/checks/checks.md#check_model_names): Models must have a name that matches the supplied regex.
+
+**Sources**
+
+* [`check_source_has_meta_keys`](./dbt_bouncer/checks/checks.md#check_source_has_meta_keys): The `meta` config for sources must have the specified keys.
+
+# Development
+
+To setup your development environment, fork this repository and run:
+
+```bash
+poetry shell
+poetry install
+pre-commit install
+```
+
+Set required environment variables by copying `.env.example` to `.env` and updating the values.
+
+All tests can be run via:
+```bash
+make build-artifacts # Rebuilds dbt artifacts used by pytest
+make test
+```

dbt_bouncer-0.14.0/dbt_bouncer/checks/checks.md

@@ -0,0 +1,168 @@
+# `check_macro_arguments_description_populated`
+
+Macro arguments must have a populated description.
+
+**Argument(s)**:
+
+* `include`: (Optional) Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
+
+**Example**:
+```yaml
+checks:
+    - name: check_macro_arguments_description_populated
+```
+
+**Required artifact(s)**:
+
+* manifest.json
+
+---
+
+# `check_macro_description_populated`
+
+Macros must have a populated description.
+
+**Argument(s)**:
+
+* `include`: (Optional) Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
+
+**Example**:
+```yaml
+checks:
+    - name: check_macro_description_populated
+```
+
+**Required artifacts(s)**:
+
+* manifest.json
+
+---
+
+# `check_macro_name_matches_file_name`
+
+Macros names must be the same as the file they are contained in.
+
+Generic tests are also macros, however to document these tests the "name" value must be precededed with "test_".
+
+**Argument(s)**:
+
+* `include`: (Optional) Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
+
+**Example**:
+```yaml
+checks:
+    - name: check_macro_name_matches_file_name
+```
+
+**Required artifacts(s)**:
+
+* manifest.json
+
+---
+
+# `check_model_description_populated`
+
+Models must have a populated description.
+
+**Argument(s)**:
+
+* `include`: (Optional) Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
+
+**Example**:
+```yaml
+checks:
+    - name: check_model_description_populated
+```
+
+**Required artifacts(s)**:
+
+* manifest.json
+
+---
+
+# `check_model_names`
+
+Models must have a name that matches the supplied regex.
+
+**Argument(s)**:
+
+* `include`: (Optional) Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
+* `model_name_pattern`: Regex pattern to match the model name.
+
+**Example**:
+```yaml
+checks:
+    - name: check_model_names
+        include: ^intermediate
+        model_name_pattern: ^int_
+    - name: check_model_names
+        include: ^staging
+        model_name_pattern: ^stg_
+```
+
+**Required artifacts(s)**:
+
+* manifest.json
+
+---
+
+# `check_project_name`
+
+Enforce that the name of the dbt project matches a supplied regex. Generally used to enforce that project names conform to something like  `company_<DOMAIN>`.
+
+**Argument(s)**:
+
+* `include`: (Optional) Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
+* `project_name_pattern`: Regex pattern to match the project name.
+
+**Example**:
+```yaml
+checks:
+    - name: check_project_name
+```
+
+**Required artifacts(s)**:
+
+* manifest.json
+
+---
+
+# `check_source_has_meta_keys`
+
+The `meta` config for sources must have the specified keys.
+
+**Argument(s)**:
+
+* `include`: (Optional) Regex pattern to match the macro path. Only macro paths that match the pattern will be checked.
+* `key`:
+
+**Example**:
+```yaml
+checks:
+    - name: check_source_has_meta_keys
+      keys:
+        - contact:
+            - email
+            - slack
+        - owner
+```
+
+**Required artifacts(s)**:
+
+* manifest.json
+
+---
+
+# `check_top_level_directories`
+
+Only specified top-level directories are allowed to contain models.
+
+**Example**:
+```yaml
+checks:
+    - name: check_top_level_directories
+```
+
+**Required artifacts(s)**:
+
+* manifest.json

dbt_bouncer-0.14.0/dbt_bouncer/checks/conftest.py

@@ -0,0 +1,20 @@
+def pytest_configure(config):
+    config.addinivalue_line("markers", "iterate_over_macros: Tests that should run once per macro")
+    config.addinivalue_line(
+        "markers",
+        "iterate_over_models: Tests that should run once per model",
+    )
+    config.addinivalue_line(
+        "markers",
+        "iterate_over_sources: Tests that should run once per source",
+    )
+
+    ini_values = {
+        "python_classes": ["check_", "Check"],
+        "python_files": ["check_*.py"],
+        "python_functions": ["check_*"],
+    }
+
+    for name, values in ini_values.items():
+        current = config.getini(name)
+        current[:] = values

dbt_bouncer-0.14.0/dbt_bouncer/checks/manifest/check_macros.py

@@ -0,0 +1,47 @@
+import pytest
+
+from dbt_bouncer.utils import get_check_inputs
+
+
+@pytest.mark.iterate_over_macros
+def check_macro_arguments_description_populated(request, check_config=None, macro=None) -> None:
+    """
+    Macro arguments must have a populated description.
+    """
+
+    macro = get_check_inputs(check_config=check_config, macro=macro, request=request)["macro"]
+    for arg in macro["arguments"]:
+        assert (
+            len(arg["description"].strip()) > 4
+        ), f"Argument {arg['name']} in {macro['unique_id']} does not have a populated description."
+
+
+@pytest.mark.iterate_over_macros
+def check_macro_description_populated(request, check_config=None, macro=None) -> None:
+    """
+    Macros must have a populated description.
+    """
+
+    macro = get_check_inputs(check_config=check_config, macro=macro, request=request)["macro"]
+    assert (
+        len(macro["description"].strip()) > 4
+    ), f"{macro['unique_id']} does not have a populated description."
+
+
+@pytest.mark.iterate_over_macros
+def check_macro_name_matches_file_name(request, check_config=None, macro=None) -> None:
+    """
+    Macros names must be the same as the file they are contained in.
+
+    Generic tests are also macros, however to document these tests the "name" value must be precededed with "test_".
+    """
+
+    macro = get_check_inputs(macro=macro, request=request)["macro"]
+    if macro["name"].startswith("test_"):
+        assert (
+            macro["name"][5:] == macro["path"].split("/")[-1].split(".")[0]
+        ), f"{macro['unique_id']} is not in a file named `{macro['name'][5:]}.sql`."
+    else:
+        assert (
+            macro["name"] == macro["path"].split("/")[-1].split(".")[0]
+        ), f"{macro['unique_id']} is not in a file of the same name."

dbt_bouncer-0.14.0/dbt_bouncer/checks/manifest/check_metadata.py

@@ -0,0 +1,18 @@
+import re
+
+from dbt_bouncer.utils import get_check_inputs
+
+
+def check_project_name(manifest_obj, request, check_config=None):
+    """
+    Enforce that the name of the dbt project matches a supplied regex. Generally used to enforce that project names conform to something like  `company_<DOMAIN>`.
+    """
+
+    check_config = get_check_inputs(check_config=check_config, request=request)["check_config"]
+
+    assert (
+        re.compile(check_config["project_name_pattern"].strip()).match(
+            manifest_obj.metadata.project_name
+        )
+        is not None
+    ), f"Project name (`{manifest_obj.metadata.project_name}`) does not conform to the supplied regex `({check_config['project_name_pattern'].strip()})`."

dbt_bouncer-0.14.0/dbt_bouncer/checks/manifest/check_models.py

@@ -0,0 +1,33 @@
+import re
+
+import pytest
+
+from dbt_bouncer.utils import get_check_inputs
+
+
+@pytest.mark.iterate_over_models
+def check_model_description_populated(request, check_config=None, model=None):
+    """
+    Models must have a populated description.
+    """
+
+    model = get_check_inputs(model=model, request=request)["model"]
+
+    assert (
+        len(model["description"].strip()) > 4
+    ), f"{model['unique_id']} does not have a populated description."
+
+
+@pytest.mark.iterate_over_models
+def check_model_names(request, check_config=None, model=None):
+    """
+    Models must have a name that matches the supplied regex.
+    """
+
+    input_vars = get_check_inputs(check_config=check_config, model=model, request=request)
+    check_config = input_vars["check_config"]
+    model = input_vars["model"]
+
+    assert (
+        re.compile(check_config["model_name_pattern"].strip()).match(model["name"]) is not None
+    ), f"{model['unique_id']} does not match the supplied regex `({check_config['model_name_pattern'].strip()})`."

dbt_bouncer-0.14.0/dbt_bouncer/checks/manifest/check_project_directories.py

@@ -0,0 +1,18 @@
+import pytest
+
+from dbt_bouncer.utils import get_check_inputs
+
+
+@pytest.mark.iterate_over_models
+def check_top_level_directories(request, model=None):
+    """
+    Only specified top-level directories are allowed to contain models.
+    """
+
+    model = get_check_inputs(model=model, request=request)["model"]
+    top_level_dir = model["path"].split("/")[0]
+    assert top_level_dir in [
+        "staging",
+        "intermediate",
+        "marts",
+    ], f"{model['unique_id']} is located in `{top_level_dir}`, this is not a valid top-level directory."