dbt-autofix 0.1.0__tar.gz

dbt_autofix-0.1.0/PKG-INFO

@@ -0,0 +1,69 @@
+Metadata-Version: 2.1
+Name: dbt-autofix
+Version: 0.1.0
+Summary: CLI to autofix deprecations in dbt projects
+Requires-Python: >=3.9
+Requires-Dist: pyyaml>=6.0.2
+Requires-Dist: typer>=0.15.2
+Requires-Dist: yamllint>=1.37.0
+Requires-Dist: rich>=13.7.0
+Requires-Dist: click<8.2.0
+Requires-Dist: ruamel-yaml>=0.18.10
+Requires-Dist: ruamel-yaml-string>=0.1.1
+Requires-Dist: httpx>=0.28.1
+Provides-Extra: test
+Requires-Dist: pytest>=7.2.0; extra == "test"
+Requires-Dist: pytest-cov>=4.1.0; extra == "test"
+Requires-Dist: pre-commit>=4.2.0; extra == "test"
+Requires-Dist: nox; extra == "test"
+Description-Content-Type: text/markdown
+
+# dbt-autofix (previously dbt-cleanup)
+
+This tool can help teams clean up their dbt projects so that it conforms with dbt configuration best practices and update deprecated config:
+
+
+| Deprecation Code in dbt Core       | Files             | Description                                                                                      |
+| ---------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------ |
+| `CustomKeyInObjectDeprecation` (*) | YAML files        | Move all models configs under `config:` in YAML files                                            |
+| `CustomKeyInObjectDeprecation` (*) | YAML files        | Move all models extra config (not valid or custom) under `meta:` and `meta` under `config:`      |
+| `DuplicateYAMLKeysDeprecation`     | YAML files        | Remove duplicate keys in YAML files, keeping the second one to keep the same behaviour           |
+| `UnexpectedJinjaBlockDeprecation`  | SQL files         | Remove extra `{% endmacro %}` and `{% endif %}` that don't have corresponding opening statements |
+| -                                  | `dbt_project.yml` | Prefix all configs for modeles/tests etc... with a `+`                                           |
+| `ConfigDataPathDeprecation`        | `dbt_project.yml` | Remove deprecated config for data path (now seed)                                                |
+| `ConfigLogPathDeprecation`         | `dbt_project.yml` | Remove deprecated config for log path                                                            |
+| `ConfigSourcePathDeprecation`      | `dbt_project.yml` | Remove deprecated config for source path                                                         |
+| `ConfigTargetPathDeprecation`      | `dbt_project.yml` | Remove deprecated config for target path                                                         |
+
+(*) : those autofix rules are currently deactivated in `main` 
+
+## Installation
+
+To run it from the git repo directly, install `uv` and then
+
+run the tool directly
+```sh
+uvx --from git+https://github.com/dbt-labs/dbt-autofix.git dbt-autofix --help
+```
+
+or install it so that it can be run with `dbt-cleanup` in the future
+```sh
+uv tool install --from git+https://github.com/dbt-labs/dbt-autofix.git dbt-autofix
+```
+
+## Usage
+
+- `dbt-autofix deprecations`: refactor YAML and SQL files to fix some deprecations
+  - add `--path <mypath>` to configure the path of the dbt project (defaults to `.`)
+  - add `--dry-run` for running in dry run mode
+  - add `--json` to get resulting data in a JSONL format
+
+Each JSON object will have the following keys:
+
+- "mode": "applied" or "dry_run" 
+- "file_path": the full path of the file modified. Each file will appear only once
+- "refactors": the list of refactoring rules applied
+
+Calling `deprecations` without `--dry-run` should be safe if your dbt code is part of a git repo. 
+
+Please review the suggested changes to your dbt project before merging to `main` and make those changes go through your typical CI/CD process.

dbt_autofix-0.1.0/README.md

@@ -0,0 +1,49 @@
+# dbt-autofix (previously dbt-cleanup)
+
+This tool can help teams clean up their dbt projects so that it conforms with dbt configuration best practices and update deprecated config:
+
+
+| Deprecation Code in dbt Core       | Files             | Description                                                                                      |
+| ---------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------ |
+| `CustomKeyInObjectDeprecation` (*) | YAML files        | Move all models configs under `config:` in YAML files                                            |
+| `CustomKeyInObjectDeprecation` (*) | YAML files        | Move all models extra config (not valid or custom) under `meta:` and `meta` under `config:`      |
+| `DuplicateYAMLKeysDeprecation`     | YAML files        | Remove duplicate keys in YAML files, keeping the second one to keep the same behaviour           |
+| `UnexpectedJinjaBlockDeprecation`  | SQL files         | Remove extra `{% endmacro %}` and `{% endif %}` that don't have corresponding opening statements |
+| -                                  | `dbt_project.yml` | Prefix all configs for modeles/tests etc... with a `+`                                           |
+| `ConfigDataPathDeprecation`        | `dbt_project.yml` | Remove deprecated config for data path (now seed)                                                |
+| `ConfigLogPathDeprecation`         | `dbt_project.yml` | Remove deprecated config for log path                                                            |
+| `ConfigSourcePathDeprecation`      | `dbt_project.yml` | Remove deprecated config for source path                                                         |
+| `ConfigTargetPathDeprecation`      | `dbt_project.yml` | Remove deprecated config for target path                                                         |
+
+(*) : those autofix rules are currently deactivated in `main` 
+
+## Installation
+
+To run it from the git repo directly, install `uv` and then
+
+run the tool directly
+```sh
+uvx --from git+https://github.com/dbt-labs/dbt-autofix.git dbt-autofix --help
+```
+
+or install it so that it can be run with `dbt-cleanup` in the future
+```sh
+uv tool install --from git+https://github.com/dbt-labs/dbt-autofix.git dbt-autofix
+```
+
+## Usage
+
+- `dbt-autofix deprecations`: refactor YAML and SQL files to fix some deprecations
+  - add `--path <mypath>` to configure the path of the dbt project (defaults to `.`)
+  - add `--dry-run` for running in dry run mode
+  - add `--json` to get resulting data in a JSONL format
+
+Each JSON object will have the following keys:
+
+- "mode": "applied" or "dry_run" 
+- "file_path": the full path of the file modified. Each file will appear only once
+- "refactors": the list of refactoring rules applied
+
+Calling `deprecations` without `--dry-run` should be safe if your dbt code is part of a git repo. 
+
+Please review the suggested changes to your dbt project before merging to `main` and make those changes go through your typical CI/CD process.

dbt_autofix-0.1.0/dbt_autofix/duplicate_keys.py

@@ -0,0 +1,127 @@
+from dataclasses import dataclass
+from pathlib import Path
+from typing import List, Tuple
+
+import yaml
+import yamllint.config
+import yamllint.linter
+from rich.console import Console
+
+from dbt_autofix.refactor import DbtYAML
+
+console = Console()
+
+config = """
+rules:
+  key-duplicates: enable
+"""
+
+yaml_config = yamllint.config.YamlLintConfig(config)
+
+
+@dataclass
+class DuplicateFound:
+    file: Path
+    line: int
+    key: str
+    value: str
+
+    def __str__(self):
+        return f"{self.file}:{self.line} -- {self.value}"
+
+
+def find_duplicate_keys(
+    root_dir: Path,
+    dry_run: bool = False,
+) -> Tuple[List[DuplicateFound], List[DuplicateFound]]:
+    """
+    Find duplicate keys in the project and packages.
+    """
+    project_duplicates: List[DuplicateFound] = []
+    package_duplicates: List[DuplicateFound] = []
+
+    yml_files = set(root_dir.glob("**/*.yml")).union(set(root_dir.glob("**/*.yaml")))
+    yml_files_target = set((root_dir / "target").glob("**/*.yml")).union(set((root_dir / "target").glob("**/*.yaml")))
+
+    packages_path = yaml.safe_load((root_dir / "dbt_project.yml").read_text()).get(
+        "packages-install-path", "dbt_packages"
+    )
+
+    yml_files_packages = set((root_dir / packages_path).glob("**/*.yml")).union(
+        set((root_dir / packages_path).glob("**/*.yaml"))
+    )
+
+    # this is a hack to avoid checking integration_tests. it won't work everywhere but it's good enough for now
+    yml_files_packages_integration_tests = set((root_dir / packages_path).glob("**/integration_tests/**/*.yml")).union(
+        set((root_dir / packages_path).glob("**/integration_tests/**/*.yaml"))
+    )
+    yml_files_packages_not_integration_tests = yml_files_packages - yml_files_packages_integration_tests
+
+    yml_files_not_target_or_packages = yml_files - yml_files_target - yml_files_packages
+
+    # Check project YML files
+    for file in yml_files_not_target_or_packages:
+        file_with_duplicate = False
+        file_content = file.read_text()
+        for p in yamllint.linter.run(file_content, yaml_config):
+            if p.rule == "key-duplicates":
+                file_with_duplicate = True
+                project_duplicates.append(
+                    DuplicateFound(
+                        file=file,
+                        line=p.line,
+                        key=p.desc.split('"')[1] if '"' in p.desc else "",  # Extract key from description
+                        value=p.desc,
+                    )
+                )
+        if file_with_duplicate and not dry_run:
+            without_duplicates = yaml.safe_load(file_content)
+            ruamel_yaml = DbtYAML()
+            ruamel_yaml.dump_to_string(without_duplicates)  # type: ignore
+
+    # Check package YML files
+    for file in yml_files_packages_not_integration_tests:
+        file_content = file.read_text()
+        for p in yamllint.linter.run(file_content, yaml_config):
+            if p.rule == "key-duplicates":
+                package_duplicates.append(
+                    DuplicateFound(
+                        file=file,
+                        line=p.line,
+                        key=p.desc.split('"')[1] if '"' in p.desc else "",  # Extract key from description
+                        value=p.desc,
+                    )
+                )
+
+    return project_duplicates, package_duplicates
+
+
+def print_duplicate_keys(project_duplicates: List[DuplicateFound], package_duplicates: List[DuplicateFound]) -> None:
+    """
+    Print duplicate keys in the project and packages as well as instructions to fix them.
+    """
+    if not project_duplicates and not package_duplicates:
+        return
+
+    if project_duplicates:
+        console.print("\nThere are issues in your project YML files", style="bold red")
+        console.print(
+            (
+                "Please remove duplicates by hand. dbt's default behavior is to keep the last occurence of a key.\n"
+                "If you want to keep the same behaviour remove or comments lines found for the same key and before in the file.\n"
+                "Once you have done all the changes in the files, run the tool again.\n"
+            )
+        )
+        for dup in project_duplicates:
+            console.print(str(dup))
+
+    if package_duplicates:
+        console.print(
+            (
+                "\nThose packages might have issues. If those are not maintained by you, check if there are updates available. "
+                "If they are private packages, remove duplicates in their own repository and publish a new version.\n"
+            ),
+            style="bold red",
+        )
+        for dup in package_duplicates:
+            console.print(str(dup))

dbt_autofix-0.1.0/dbt_autofix/fields_properties_configs.py

@@ -0,0 +1,328 @@
+import csv
+from dataclasses import dataclass
+
+from rich import print
+
+
+@dataclass
+class AllowedConfig:
+    allowed_config_fields: set[str]
+    allowed_properties: set[str]
+
+    def __post_init__(self):
+        self.allowed_config_fields_without_meta = self.allowed_config_fields - {"meta"}
+        # in case we forgot to remove meta from the allowed properties
+        self.allowed_properties = self.allowed_properties - {"meta"}
+
+
+models_allowed_config = AllowedConfig(
+    allowed_config_fields=set(
+        [
+            # model specific
+            "materialized",
+            "sql_header",
+            "on_configuration_change",
+            "unique_key",
+            "incremental_strategy",  # was missing at first
+            "on_schema_change",  # was missing at first
+            "batch_size",
+            "begin",
+            "lookback",
+            "concurrent_batches",
+            # general
+            "enabled",
+            "tags",
+            "pre_hook",
+            "post_hook",
+            "database",
+            "schema",
+            "alias",
+            "persist_docs",
+            "meta",
+            "grants",
+            "contract",
+            "event_time",
+            # Snowflake: https://github.com/dbt-labs/dbt-adapters/blob/af33935b119347cc021554ea854884bce986ef8d/dbt-snowflake/src/dbt/adapters/snowflake/impl.py#L42-L58
+            "transient",
+            "cluster_by",
+            "automatic_clustering",
+            "secure",
+            "copy_grants",
+            "snowflake_warehouse",
+            "query_tag",
+            "tmp_relation_type",
+            "merge_update_columns",
+            "target_lag",
+            "table_format",
+            "external_volume",
+            "base_location_root",
+            "base_location_subpath",
+            # BQ: https://github.com/dbt-labs/dbt-adapters/blob/af33935b119347cc021554ea854884bce986ef8d/dbt-bigquery/src/dbt/adapters/bigquery/impl.py#L98
+            "dataset",
+            "project",
+            "cluster_by",
+            "partition_by",
+            "kms_key_name",
+            "labels",
+            "partitions",
+            "grant_access_to",
+            "hours_to_expiration",
+            "require_partition_filter",
+            "partition_expiration_days",
+            "merge_update_columns",
+            "enable_refresh",
+            "refresh_interval_minutes",
+            "max_staleness",
+            "enable_list_inference",
+            "intermediate_format",
+            "submission_method",
+            # Postgres
+            "unlogged",
+            "indexes",
+            # Redshift
+            "sort_type",
+            "dist",
+            "sort",
+            "bind",
+            "backup",
+            "auto_refresh",
+            # DBX: https://github.com/databricks/dbt-databricks/blob/fde68588d45e8a299a83318acdca6c97081088e8/dbt/adapters/databricks/impl.py#L105
+            "file_format",
+            "table_format",
+            "location_root",
+            "include_full_name_in_path",
+            "partition_by",
+            "clustered_by",
+            "liquid_clustered_by",
+            "auto_liquid_cluster",
+            "buckets",
+            "options",
+            "merge_update_columns",
+            "merge_exclude_columns",
+            "databricks_tags",
+            "tblproperties",
+            "zorder",
+            "unique_tmp_table_suffix",
+            "skip_non_matched_step",
+            "skip_matched_step",
+            "matched_condition",
+            "not_matched_condition",
+            "not_matched_by_source_action",
+            "not_matched_by_source_condition",
+            "target_alias",
+            "source_alias",
+            "merge_with_schema_evolution",
+            # moved from property
+            "docs",
+            "access",
+            "group",
+            # new
+            "freshness",
+        ]
+    ),
+    allowed_properties=set(
+        [
+            "name",
+            "description",
+            "latest_version",
+            "deprecation_date",
+            "config",
+            "constraints",
+            "data_tests",
+            "tests",
+            "columns",
+            "time_spine",
+            "versions",
+        ]
+    ),
+)
+sources_allowed_config = AllowedConfig(
+    allowed_config_fields=set(["enabled", "event_time", "meta", "freshness", "tags"]),
+    allowed_properties=set(
+        [
+            "name",
+            "description",
+            "database",
+            "schema",
+            "loader",
+            "loaded_at_field",
+            "quoting",
+            "tables",
+            "config",
+        ]
+    ),
+)
+snapshots_allowed_config = AllowedConfig(
+    allowed_config_fields=set(
+        [
+            # snapshot specific
+            "database",
+            "schema",
+            "unique_key",
+            "strategy",
+            "updated_at",
+            "check_cols",
+            "snapshot_meta_column_names",
+            "hard_deletes",
+            "dbt_valid_to_current",
+            # general
+            "enabled",
+            "tags",
+            "alias",
+            "pre_hook",
+            "post_hook",
+            "persist_docs",
+            "grants",
+            "event_time",
+            # moved from property
+            "docs",
+            # not in docs for config
+            "meta",
+        ]
+    ),
+    allowed_properties=set(
+        [
+            "name",
+            "description",
+            "config",
+            "tests",
+            "columns",
+        ]
+    ),
+)
+seeds_allowed_config = AllowedConfig(
+    allowed_config_fields=set(
+        [
+            # seed specific
+            "quote_columns",
+            "column_types",
+            "delimiter",
+            # general
+            "enabled",
+            "tags",
+            "pre_hook",
+            "post_hook",
+            "database",
+            "schema",
+            "alias",
+            "persist_docs",
+            "full_refresh",
+            "meta",
+            "grants",
+            "event_time",
+            # moved from property
+            "docs",
+        ]
+    ),
+    allowed_properties=set(
+        [
+            "name",
+            "description",
+            "config",
+            "tests",
+            "columns",
+        ]
+    ),
+)
+
+tests_allowed_config = AllowedConfig(
+    allowed_config_fields=set(
+        [
+            # general
+            "enabled",
+            "tags",
+            "meta",
+            "database",
+            "schema",
+            "alias",
+            # data tests
+            "fail_calc",
+            "limit",
+            "severity",
+            "error_if",
+            "warn_if",
+            "store_failures",
+            "where",
+        ]
+    ),
+    allowed_properties=set(
+        [
+            "name",
+            "description",
+            "config",
+        ]
+    ),
+)
+
+fields_per_node_type = {
+    "models": models_allowed_config,
+    "seeds": seeds_allowed_config,
+    "tests": tests_allowed_config,
+    "sources": sources_allowed_config,
+    "snapshots": snapshots_allowed_config,
+}
+
+
+def print_matrix():  # noqa: PLR0912
+    from dbt_autofix.retrieve_schemas import yaml_specs_per_node_type
+
+    results = dict()
+    for node_type, fields_config in fields_per_node_type.items():
+        allowed_config_fields = fields_config.allowed_config_fields
+        for field in allowed_config_fields:
+            if field not in results:
+                results[field] = {f"{node_type}-bestguess": "config"}
+            else:
+                results[field].update({f"{node_type}-bestguess": "config"})
+
+        allowed_properties = fields_config.allowed_properties
+        for property in allowed_properties:
+            if property not in results:
+                results[property] = {f"{node_type}-bestguess": "property"}
+            else:
+                results[property].update({f"{node_type}-bestguess": "property"})
+
+    for node_type, fields_config in yaml_specs_per_node_type.items():
+        allowed_config_fields = fields_config.allowed_config_fields
+        for field in allowed_config_fields:
+            if field not in results:
+                results[field] = {f"{node_type}-fusion": "config"}
+            else:
+                results[field].update({f"{node_type}-fusion": "config"})
+
+        allowed_properties = fields_config.allowed_properties
+        for property in allowed_properties:
+            if property not in results:
+                results[property] = {f"{node_type}-fusion": "property"}
+            else:
+                results[property].update({f"{node_type}-fusion": "property"})
+
+    # Assuming your dictionary is named 'results'
+    with open("config_resources.csv", "w", newline="") as csvfile:
+        # Determine all possible resource types from the data
+        resource_types = sorted(
+            set(resource_type for config_values in results.values() for resource_type in config_values.keys())
+        )
+
+        # Create fieldnames with 'field_name' as the first column followed by all resource types
+        fieldnames = ["field_name", *resource_types]
+
+        # Create CSV writer
+        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
+
+        # Write the header
+        writer.writeheader()
+
+        # Sort field_names alphabetically
+        sorted_field_names = sorted(results.keys())
+
+        # Write each row
+        for field_name in sorted_field_names:
+            resources = results[field_name]
+            row = {"field_name": field_name}
+            # Add values for each resource type
+            for resource_type in resource_types:
+                row[resource_type] = resources.get(resource_type, "")
+            writer.writerow(row)
+
+    print("CSV file 'config_resources.csv' has been created successfully.")

dbt_autofix-0.1.0/dbt_autofix/main.py

@@ -0,0 +1,60 @@
+from pathlib import Path
+
+import typer
+from rich import print
+from rich.console import Console
+from typing_extensions import Annotated
+
+from dbt_autofix.duplicate_keys import find_duplicate_keys, print_duplicate_keys
+from dbt_autofix.fields_properties_configs import print_matrix
+from dbt_autofix.refactor import apply_changesets, changeset_all_sql_yml_files
+
+console = Console()
+error_console = Console(stderr=True)
+
+app = typer.Typer(
+    help="A tool to help clean up dbt projects",
+    no_args_is_help=True,
+    add_completion=False,
+)
+
+current_dir = Path.cwd()
+
+
+@app.command(name="list-yaml-duplicates")
+def identify_duplicate_keys(
+    path: Annotated[Path, typer.Option("--path", "-p", help="The path to the dbt project")] = current_dir,
+):
+    print(f"[green]Identifying duplicates in {path}[/green]\n")
+    project_duplicates, package_duplicates = find_duplicate_keys(path)
+    print_duplicate_keys(project_duplicates, package_duplicates)
+
+
+@app.command(name="deprecations")
+def refactor_yml(
+    path: Annotated[Path, typer.Option("--path", "-p", help="The path to the dbt project")] = current_dir,
+    dry_run: Annotated[bool, typer.Option("--dry-run", "-d", help="In dry run mode, do not apply changes")] = False,
+    json_output: Annotated[bool, typer.Option("--json", "-j", help="Output in JSON format")] = False,
+):
+    changesets = changeset_all_sql_yml_files(path, dry_run)
+    yaml_results, sql_results = changesets
+    if dry_run:
+        if not json_output:
+            error_console.print("[red]-- Dry run mode, not applying changes --[/red]")
+        for changeset in yaml_results:
+            if changeset.refactored:
+                changeset.print_to_console(json_output)
+        for changeset in sql_results:
+            if changeset.refactored:
+                changeset.print_to_console(json_output)
+    else:
+        apply_changesets(yaml_results, sql_results, json_output)
+
+
+@app.command(hidden=True)
+def print_fields_matrix():
+    print_matrix()
+
+
+if __name__ == "__main__":
+    app()