airflow-dbt-python 2.0.1__tar.gz → 2.2.0__tar.gz

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/PKG-INFO

@@ -1,24 +1,19 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.3
 Name: airflow-dbt-python
-Version: 2.0.1
+Version: 2.2.0
 Summary: A collection of Airflow operators, hooks, and utilities to execute dbt commands
-Home-page: https://github.com/tomasfarias/airflow-dbt-python
 License: MIT
 Author: Tomás Farías Santana
 Author-email: tomas@tomasfarias.dev
-Requires-Python: >=3.8,<3.12
+Requires-Python: >=3.9,<3.13
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.12
 Provides-Extra: adapters
 Provides-Extra: airflow-providers
 Provides-Extra: bigquery
@@ -29,23 +24,17 @@ Provides-Extra: redshift
 Provides-Extra: s3
 Provides-Extra: snowflake
 Provides-Extra: spark
-Requires-Dist: apache-airflow (>=2.2)
+Requires-Dist: apache-airflow (>=2.8)
 Requires-Dist: apache-airflow-providers-amazon (>=3.0.0) ; extra == "airflow-providers" or extra == "s3"
 Requires-Dist: apache-airflow-providers-ssh (>=3.0.0) ; extra == "airflow-providers" or extra == "git"
 Requires-Dist: connexion (<3.0)
 Requires-Dist: contextlib-chdir (==1.0.2) ; python_version < "3.11"
-Requires-Dist: dbt-bigquery (>=1.4.0,<1.6.0) ; (python_version < "3.8") and (extra == "adapters" or extra == "bigquery")
-Requires-Dist: dbt-bigquery (>=1.4.0,<1.8.0) ; (python_version >= "3.11") and (extra == "adapters" or extra == "bigquery")
-Requires-Dist: dbt-core (>=1.4.0,<1.6.0) ; python_version < "3.8"
-Requires-Dist: dbt-core (>=1.4.0,<1.8.0) ; python_version >= "3.11"
-Requires-Dist: dbt-postgres (>=1.4.0,<1.6.0) ; (python_version < "3.8") and (extra == "adapters" or extra == "postgres")
-Requires-Dist: dbt-postgres (>=1.4.0,<1.8.0) ; (python_version >= "3.11") and (extra == "adapters" or extra == "postgres")
-Requires-Dist: dbt-redshift (>=1.4.0,<1.6.0) ; (python_version < "3.8") and (extra == "adapters" or extra == "redshift")
-Requires-Dist: dbt-redshift (>=1.4.0,<1.8.0) ; (python_version >= "3.11") and (extra == "adapters" or extra == "redshift")
-Requires-Dist: dbt-snowflake (>=1.4.0,<1.6.0) ; (python_version < "3.8") and (extra == "adapters" or extra == "snowflake")
-Requires-Dist: dbt-snowflake (>=1.4.0,<1.8.0) ; (python_version >= "3.11") and (extra == "adapters" or extra == "snowflake")
-Requires-Dist: dbt-spark (>=1.4.0,<1.6.0) ; (python_version < "3.8") and (extra == "adapters" or extra == "spark")
-Requires-Dist: dbt-spark (>=1.4.0,<1.8.0) ; (python_version >= "3.11") and (extra == "adapters" or extra == "spark")
+Requires-Dist: dbt-bigquery (>=1.8.0,<2.0.0) ; extra == "adapters" or extra == "bigquery"
+Requires-Dist: dbt-core (>=1.8.0,<2.0.0)
+Requires-Dist: dbt-postgres (>=1.8.0,<2.0.0) ; extra == "adapters" or extra == "postgres"
+Requires-Dist: dbt-redshift (>=1.8.0,<2.0.0) ; extra == "adapters" or extra == "redshift"
+Requires-Dist: dbt-snowflake (>=1.8.0,<2.0.0) ; extra == "adapters" or extra == "snowflake"
+Requires-Dist: dbt-spark (>=1.8.0,<2.0.0) ; extra == "adapters" or extra == "spark"
 Requires-Dist: dulwich (>=0.21) ; extra == "git"
 Project-URL: Repository, https://github.com/tomasfarias/airflow-dbt-python
 Description-Content-Type: text/markdown
@@ -67,10 +56,10 @@ Read the [documentation](https://airflow-dbt-python.readthedocs.io) for examples
 ## Requirements
 
 Before using *airflow-dbt-python*, ensure you meet the following requirements:
-* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.4.0 or later.
-* An Airflow environment using version 2.2 or later.
+* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.8 or later.
+* An Airflow environment using version 2.8 or later.
 
-  * If using any managed service, like AWS MWAA, ensure your environment is created with a supported version of Airflow.
+  * If using any managed service, like AWS MWAA or GCP Cloud Composer 2/3, ensure your environment is created with a supported version of Airflow.
   * If self-hosting, Airflow installation instructions can be found in their [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).
 
 * Running Python 3.8 or later in your Airflow environment.
@@ -81,7 +70,7 @@ Before using *airflow-dbt-python*, ensure you meet the following requirements:
 
 > **Note**
 >
-> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/).
+> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) and [GCP Cloud Composer 2/3](https://aws.amazon.com/managed-workflows-for-apache-airflow/).
 
 ## From PyPI
 
@@ -118,6 +107,12 @@ Add *airflow-dbt-python* to your `requirements.txt` file and edit your Airflow e
 
 Read the [documentation](https://airflow-dbt-python.readthedocs.io/en/latest/getting_started.html#installing-in-mwaa) for more a more detailed AWS MWAA installation breakdown.
 
+## In GCP Cloud Composer
+
+Add *airflow-dbt-python* to your PyPI packages list.
+
+Read the [documentation](https://cloud.google.com/composer/docs/composer-2/install-python-dependencies#install-pypi) for more a more detailed GCP Cloud Composer 2 installation breakdown.
+
 ## In other managed services
 
 *airflow-dbt-python* should be compatible with most or all Airflow managed services. Consult the documentation specific to your provider.
@@ -171,7 +166,7 @@ See an example DAG [here](examples/airflow_connection_target_dag.py).
 
 Although [`dbt`](https://docs.getdbt.com/) is meant to be installed and used as a CLI, we may not have control of the environment where Airflow is running, disallowing us the option of using *dbt* as a CLI.
 
-This is exactly what happens when using [Amazon's Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) or MWAA: although a list of Python requirements can be passed, the CLI cannot be found in the worker's PATH.
+This is exactly what happens when using [Amazon's Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) (aka MWAA): although a list of Python requirements can be passed, the CLI cannot be found in the worker's PATH.
 
 There is a workaround which involves using Airflow's `BashOperator` and running Python from the command line:
 

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/README.md

@@ -15,10 +15,10 @@ Read the [documentation](https://airflow-dbt-python.readthedocs.io) for examples
 ## Requirements
 
 Before using *airflow-dbt-python*, ensure you meet the following requirements:
-* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.4.0 or later.
-* An Airflow environment using version 2.2 or later.
+* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.8 or later.
+* An Airflow environment using version 2.8 or later.
 
-  * If using any managed service, like AWS MWAA, ensure your environment is created with a supported version of Airflow.
+  * If using any managed service, like AWS MWAA or GCP Cloud Composer 2/3, ensure your environment is created with a supported version of Airflow.
   * If self-hosting, Airflow installation instructions can be found in their [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).
 
 * Running Python 3.8 or later in your Airflow environment.
@@ -29,7 +29,7 @@ Before using *airflow-dbt-python*, ensure you meet the following requirements:
 
 > **Note**
 >
-> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/).
+> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) and [GCP Cloud Composer 2/3](https://aws.amazon.com/managed-workflows-for-apache-airflow/).
 
 ## From PyPI
 
@@ -66,6 +66,12 @@ Add *airflow-dbt-python* to your `requirements.txt` file and edit your Airflow e
 
 Read the [documentation](https://airflow-dbt-python.readthedocs.io/en/latest/getting_started.html#installing-in-mwaa) for more a more detailed AWS MWAA installation breakdown.
 
+## In GCP Cloud Composer
+
+Add *airflow-dbt-python* to your PyPI packages list.
+
+Read the [documentation](https://cloud.google.com/composer/docs/composer-2/install-python-dependencies#install-pypi) for more a more detailed GCP Cloud Composer 2 installation breakdown.
+
 ## In other managed services
 
 *airflow-dbt-python* should be compatible with most or all Airflow managed services. Consult the documentation specific to your provider.
@@ -119,7 +125,7 @@ See an example DAG [here](examples/airflow_connection_target_dag.py).
 
 Although [`dbt`](https://docs.getdbt.com/) is meant to be installed and used as a CLI, we may not have control of the environment where Airflow is running, disallowing us the option of using *dbt* as a CLI.
 
-This is exactly what happens when using [Amazon's Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) or MWAA: although a list of Python requirements can be passed, the CLI cannot be found in the worker's PATH.
+This is exactly what happens when using [Amazon's Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) (aka MWAA): although a list of Python requirements can be passed, the CLI cannot be found in the worker's PATH.
 
 There is a workaround which involves using Airflow's `BashOperator` and running Python from the command line:
 

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/__init__.py

@@ -1,2 +1,3 @@
 """Provides an Airflow operator and hooks to run all or most dbtcommands."""
+
 from .__version__ import __author__, __copyright__, __title__, __version__

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/__version__.py

@@ -1,5 +1,6 @@
 """The module's version information."""
+
 __author__ = "Tomás Farías Santana"
 __copyright__ = "Copyright 2021 Tomás Farías Santana"
 __title__ = "airflow-dbt-python"
-__version__ = "2.0.1"
+__version__ = "2.2.0"

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/hooks/dbt.py

@@ -1,4 +1,5 @@
 """Provides a hook to interact with a dbt project."""
+
 from __future__ import annotations
 
 import json
@@ -24,14 +25,11 @@ from airflow.exceptions import AirflowException
 from airflow.hooks.base import BaseHook
 from airflow.models.connection import Connection
 
-from airflow_dbt_python.utils.version import DBT_INSTALLED_LESS_THAN_1_5
-
 if sys.version_info >= (3, 11):
     from contextlib import chdir as chdir_ctx
 else:
     from contextlib_chdir import chdir as chdir_ctx
 
-
 if TYPE_CHECKING:
     from dbt.contracts.results import RunResult
     from dbt.task.base import BaseTask
@@ -227,18 +225,11 @@ class DbtHook(BaseHook):
             A tuple containing a boolean indicating success and optionally the results
                 of running the dbt command.
         """
-        from dbt.adapters.factory import register_adapter
+        from dbt.adapters.factory import adapter_management
         from dbt.task.base import get_nearest_project_dir
         from dbt.task.clean import CleanTask
         from dbt.task.deps import DepsTask
-
-        from airflow_dbt_python.utils.version import DBT_INSTALLED_LESS_THAN_1_5
-
-        if DBT_INSTALLED_LESS_THAN_1_5:
-            from dbt.main import adapter_management, track_run  # type: ignore
-        else:
-            from dbt.adapters.factory import adapter_management
-            from dbt.tracking import track_run
+        from dbt.tracking import track_run
 
         config = self.get_dbt_task_config(command, **kwargs)
         extra_target = self.get_dbt_target_from_connection(config.target)
@@ -252,36 +243,24 @@ class DbtHook(BaseHook):
         ) as dbt_dir:
             # When creating tasks via from_args, dbt switches to the project directory.
             # We have to do that here as we are not using from_args.
-            if DBT_INSTALLED_LESS_THAN_1_5:
-                # For compatibility with older versions of dbt, as the signature
-                # of move_to_nearest_project_dir changed in dbt-core 1.5 to take
-                # just the project_dir.
-                nearest_project_dir = get_nearest_project_dir(config)  # type: ignore
-            else:
-                nearest_project_dir = get_nearest_project_dir(config.project_dir)
+            nearest_project_dir = get_nearest_project_dir(config.project_dir)
 
             with chdir_ctx(nearest_project_dir):
-                config.dbt_task.pre_init_hook(config)
                 self.ensure_profiles(config)
 
-                task, runtime_config = config.create_dbt_task(
-                    extra_target, write_perf_info
-                )
-                requires_profile = isinstance(task, (CleanTask, DepsTask))
-
-                self.setup_dbt_logging(task, config.debug)
+                with adapter_management():
+                    task, runtime_config = config.create_dbt_task(
+                        extra_target, write_perf_info
+                    )
+                    requires_profile = isinstance(task, (CleanTask, DepsTask))
 
-                if runtime_config is not None and not requires_profile:
-                    # The deps command installs the dependencies, which means they may
-                    # not exist before deps runs and the following would raise a
-                    # CompilationError.
-                    runtime_config.load_dependencies()
+                    self.setup_dbt_logging(task, config.debug)
 
-                results = None
-                with adapter_management():
-                    if not requires_profile:
-                        if runtime_config is not None:
-                            register_adapter(runtime_config)
+                    if runtime_config is not None and not requires_profile:
+                        # The deps command installs the dependencies, which means they
+                        # may not exist before deps runs and the following would raise a
+                        # CompilationError.
+                        runtime_config.load_dependencies()
 
                     with track_run(task):
                         results = task.run()
@@ -419,19 +398,11 @@ class DbtHook(BaseHook):
         default_stdout. As these are initialized by the CLI app, we need to
         initialize them here.
         """
-        from dbt.events.functions import setup_event_logger
-
-        log_path = None
-        if task.config is not None:
-            log_path = getattr(task.config, "log_path", None)
-
-        if DBT_INSTALLED_LESS_THAN_1_5:
-            setup_event_logger(log_path or "logs")
-        else:
-            from dbt.flags import get_flags
+        from dbt.events.logging import setup_event_logger
+        from dbt.flags import get_flags
 
-            flags = get_flags()
-            setup_event_logger(flags)
+        flags = get_flags()
+        setup_event_logger(flags)
 
         configured_file = logging.getLogger("configured_file")
         file_log = logging.getLogger("file_log")
@@ -458,7 +429,7 @@ class DbtHook(BaseHook):
         if not profiles_path.exists():
             profiles_path.parent.mkdir(exist_ok=True)
             with profiles_path.open("w", encoding="utf-8") as f:
-                f.write("config:\n  send_anonymous_usage_stats: false\n")
+                f.write("flags:\n  send_anonymous_usage_stats: false\n")
 
     def get_dbt_target_from_connection(
         self, target: Optional[str]

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/hooks/git.py

@@ -1,4 +1,5 @@
 """A concrete DbtRemoteHook for git repositories with dulwich."""
+
 import datetime as dt
 from typing import Callable, Optional, Tuple, Union
 
@@ -90,7 +91,7 @@ class DbtGitRemoteHook(SSHHook, DbtRemoteHook):
 
             repo.stage(str(f.relative_to(source)))
 
-        ts = dt.datetime.utcnow()
+        ts = dt.datetime.now(dt.timezone.utc)
         repo.do_commit(
             self.commit_msg.format(ts=ts).encode(), self.commit_author.encode()
         )

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/hooks/localfs.py

@@ -2,6 +2,7 @@
 
 Intended to be used only when running Airflow with a LocalExceutor.
 """
+
 from __future__ import annotations
 
 import shutil

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/hooks/remote.py

@@ -4,6 +4,7 @@ Internally, DbtRemoteHooks can use Airflow hooks to execute the actual operation
 
 Currently, only AWS S3 and the local filesystem are supported as remotes.
 """
+
 from abc import ABC, abstractmethod
 from pathlib import Path
 from typing import Optional, Type

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/hooks/s3.py

@@ -1,4 +1,5 @@
 """An implementation for an S3 remote for dbt."""
+
 from __future__ import annotations
 
 from typing import Iterable, Optional

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/operators/dbt.py

@@ -1,4 +1,5 @@
 """Airflow operators for all dbt commands."""
+
 from __future__ import annotations
 
 import datetime as dt
@@ -11,11 +12,13 @@ from airflow.exceptions import AirflowException
 from airflow.models.baseoperator import BaseOperator
 from airflow.models.xcom import XCOM_RETURN_KEY
 
+from airflow_dbt_python.utils.enums import LogFormat
+
 if TYPE_CHECKING:
     from dbt.contracts.results import RunResult
 
     from airflow_dbt_python.hooks.dbt import DbtHook, DbtTaskResult
-    from airflow_dbt_python.utils.enums import LogFormat, Output
+    from airflow_dbt_python.utils.enums import Output
 
 base_template_fields = [
     "project_dir",
@@ -73,7 +76,7 @@ class DbtBaseOperator(BaseOperator):
         log_path: Optional[str] = None,
         log_level: Optional[str] = None,
         log_level_file: Optional[str] = None,
-        log_format: Optional[LogFormat] = None,
+        log_format: LogFormat = LogFormat.DEFAULT,
         log_cache_events: Optional[bool] = False,
         quiet: Optional[bool] = None,
         no_print: Optional[bool] = None,
@@ -93,8 +96,8 @@ class DbtBaseOperator(BaseOperator):
         no_version_check: Optional[bool] = None,
         write_json: Optional[bool] = None,
         write_perf_info: Optional[bool] = None,
-        anonymous_usage_stats: Optional[bool] = None,
-        no_anonymous_usage_stats: Optional[bool] = None,
+        send_anonymous_usage_stats: Optional[bool] = None,
+        no_send_anonymous_usage_stats: Optional[bool] = None,
         # Extra features configuration
         dbt_conn_id: Optional[str] = "dbt_conn_id",
         profiles_conn_id: Optional[str] = None,
@@ -138,8 +141,8 @@ class DbtBaseOperator(BaseOperator):
         self.static_parser = static_parser
         self.no_static_parser = no_static_parser
 
-        self.anonymous_usage_stats = anonymous_usage_stats
-        self.no_anonymous_usage_stats = no_anonymous_usage_stats
+        self.send_anonymous_usage_stats = send_anonymous_usage_stats
+        self.no_send_anonymous_usage_stats = no_send_anonymous_usage_stats
 
         self.partial_parse = partial_parse
         self.no_partial_parse = no_partial_parse
@@ -671,13 +674,13 @@ def run_result_factory(data: list[tuple[Any, Any]]):
     We need to handle dt.datetime and agate.table.Table.
     The rest of the types should already be JSON-serializable.
     """
-    from dbt.contracts.results import agate
+    from agate.table import Table
 
     d = {}
     for key, val in data:
         if isinstance(val, dt.datetime):
             val = val.isoformat()
-        elif isinstance(val, agate.table.Table):
+        elif isinstance(val, Table):
             # agate Tables have a few print methods but they offer plain
             # text representations of the table which are not very JSON
             # friendly. There is a to_json method, but I don't think

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/utils/configs.py

@@ -1,7 +1,7 @@
 """A collection of configurations to interact with dbt-core."""
+
 from __future__ import annotations
 
-import argparse
 import dataclasses
 import json
 import os
@@ -11,23 +11,24 @@ from typing import Any, Optional, Type, Union
 
 import dbt.flags as flags
 import yaml
-from dbt.adapters.factory import register_adapter
+from dbt.cli.option_types import Package
 from dbt.clients import yaml_helper  # type: ignore
 from dbt.config.profile import Profile, read_profile
 from dbt.config.project import PartialProject, Project
 from dbt.config.renderer import DbtProjectYamlRenderer, ProfileRenderer
 from dbt.config.runtime import RuntimeConfig
 from dbt.contracts.graph.manifest import Manifest
+from dbt.exceptions import DbtProjectError
 from dbt.graph.graph import Graph
-from dbt.helper_types import WarnErrorOptions
+from dbt.parser.manifest import parse_manifest
 from dbt.task.base import BaseTask, ConfiguredTask
 from dbt.task.build import BuildTask
 from dbt.task.clean import CleanTask
 from dbt.task.compile import CompileTask
 from dbt.task.debug import DebugTask
 from dbt.task.deps import DepsTask
+from dbt.task.docs.generate import GenerateTask
 from dbt.task.freshness import FreshnessTask
-from dbt.task.generate import GenerateTask
 from dbt.task.list import ListTask
 from dbt.task.run import RunTask
 from dbt.task.run_operation import RunOperationTask
@@ -35,38 +36,10 @@ from dbt.task.runnable import GraphRunnableTask
 from dbt.task.seed import SeedTask
 from dbt.task.snapshot import SnapshotTask
 from dbt.task.test import TestTask
-from dbt.tracking import initialize_from_flags as orig_initialize_from_flags
+from dbt.tracking import initialize_from_flags
 
 from airflow_dbt_python.utils.enums import FromStrEnum, LogFormat, Output
-from airflow_dbt_python.utils.version import (
-    DBT_INSTALLED_GTE_1_5,
-    DBT_INSTALLED_GTE_1_7,
-    DBT_INSTALLED_LESS_THAN_1_5,
-)
-
-if DBT_INSTALLED_LESS_THAN_1_5:
-    # dbt 1.5 removed the ParseTask as it only generated a manifest.
-    # This task is now handled by a simple function.
-    # Airflow-dbt-python should still support a ParseTask that does the same,
-    # but this will require writing our own ParseTask class.
-    # In the meantime, we treat ParseTask as CompileTask (which also generates
-    # a manifest) if running with dbt-core >= 1.5.
-    from dbt.task.parse import ParseTask  # type: ignore
-
-    # dbt 1.5 also changed the signature of initialize_from_flags, so we have
-    # this for backwards compatibility.
-    def initialize_from_flags(send_anonymous_usage_stats, profiles_dir):
-        """Call initialize_from_flags without arguments.
-
-        Added for backwards compatibility with versions of dbt < 1.5.
-        """
-        return orig_initialize_from_flags()  # type: ignore
-
-else:
-    from dbt.parser.manifest import ManifestLoader
-
-    ParseTask = CompileTask
-    initialize_from_flags = orig_initialize_from_flags
+from airflow_dbt_python.utils.version import DBT_INSTALLED_GTE_1_9
 
 
 def parse_yaml_args(args: Optional[Union[str, dict[str, Any]]]) -> dict[str, Any]:
@@ -108,16 +81,16 @@ class BaseConfig:
     threads: Optional[int] = None
     use_experimental_parser: Optional[bool] = None
     store_failures: Optional[bool] = None
-    vars: Optional[Union[str, dict[str, Any]]] = None
+    vars: Union[dict[str, Any]] = dataclasses.field(default_factory=dict)
     macro_debugging: Optional[bool] = None
     warn_error: Optional[bool] = None
     warn_error_options: dict[str, Any] = dataclasses.field(default_factory=dict)
     indirect_selection: str = "eager"
 
     # Logging
-    log_format: Optional[LogFormat] = None
-    log_format_file: Optional[LogFormat] = None
-    log_path: Optional[str] = None if DBT_INSTALLED_LESS_THAN_1_5 else "logs"
+    log_format: LogFormat = LogFormat.DEFAULT
+    log_format_file: LogFormat = LogFormat.DEFAULT
+    log_path: Optional[str] = "logs"
     log_level: str = "info"
     log_level_file: str = "none"
     record_timing_info: Optional[str] = None
@@ -154,51 +127,38 @@ class BaseConfig:
     version_check: Optional[bool] = None
     no_version_check: Optional[bool] = dataclasses.field(default=None, repr=False)
 
-    # dbt < 1.5
-    anonymous_usage_stats: Optional[bool] = None
-    no_anonymous_usage_stats: Optional[bool] = dataclasses.field(
+    send_anonymous_usage_stats: Optional[bool] = None
+    no_send_anonymous_usage_stats: Optional[bool] = dataclasses.field(
         default=None, repr=False
     )
 
-    # dbt >= 1.5
-    if DBT_INSTALLED_GTE_1_5:
-        send_anonymous_usage_stats: Optional[bool] = None
-        no_send_anonymous_usage_stats: Optional[bool] = dataclasses.field(
-            default=None, repr=False
-        )
-
-        write_json: Optional[bool] = None
-        no_write_json: Optional[bool] = dataclasses.field(default=None, repr=False)
+    include_saved_query: Optional[bool] = None
+    no_include_saved_query: Optional[bool] = dataclasses.field(default=None, repr=False)
 
-    # dbt >= 1.7
-    if DBT_INSTALLED_GTE_1_7:
-        from dbt.cli.option_types import Package
+    clean_project_files_only: Optional[bool] = None
+    no_clean_project_files_only: Optional[bool] = dataclasses.field(
+        default=None, repr=False
+    )
 
-        include_saved_query: Optional[bool] = None
-        no_include_saved_query: Optional[bool] = dataclasses.field(
-            default=None, repr=False
-        )
+    require_resource_names_without_spaces: bool = False
 
-        clean_project_files_only: Optional[bool] = None
-        no_clean_project_files_only: Optional[bool] = dataclasses.field(
-            default=None, repr=False
-        )
+    add_package: Optional[Package] = None
+    dry_run: bool = False
+    lock: bool = False
+    static: bool = False
+    upgrade: bool = False
 
-        add_package: Optional[Package] = None
-        dry_run: bool = False
-        lock: bool = False
-        static: bool = False
-        upgrade: bool = False
+    require_model_names_without_spaces: bool = False
+    source_freshness_run_project_hooks: bool = False
+    exclude_resource_types: list[str] = dataclasses.field(
+        default_factory=list, repr=False
+    )
 
     def __post_init__(self):
         """Post initialization actions for a dbt configuration."""
-        self.parsed_vars = parse_yaml_args(self.vars)
-        self.vars = yaml.dump(self.parsed_vars)
+        self.vars = parse_yaml_args(self.vars)
         self.set_mutually_exclusive_attributes()
 
-        self.send_anonymous_usage_stats = self.anonymous_usage_stats
-        self.no_send_anonymous_usage_stats = self.no_anonymous_usage_stats
-
     def set_mutually_exclusive_attributes(self):
         """Support pairs of mutually exclusive parameters.
 
@@ -219,7 +179,6 @@ class BaseConfig:
             ("populate_cache", True),
             ("static_parser", True),
             ("version_check", True),
-            ("anonymous_usage_stats", True),
             ("send_anonymous_usage_stats", True),
             ("write_json", True),
             ("include_saved_query", None),
@@ -252,40 +211,6 @@ class BaseConfig:
             item = item.lower()
         return super().__getattribute__(item)
 
-    def to_flags(self) -> argparse.Namespace:
-        """Return a Namespace to use as dbt flags."""
-        flags_dict = vars(flags.get_flags())
-        flags_dict["MACRO_DEBUGGING"] = self.macro_debugging
-        flags_dict["WARN_ERROR"] = self.warn_error
-        flags_dict["WARN_ERROR_OPTIONS"] = WarnErrorOptions(
-            include=self.warn_error_options.get("include", []),
-            exclude=self.warn_error_options.get("exclude", []),
-        )
-
-        flags_dict["PRINTER_WIDTH"] = self.printer_width
-        flags_dict["TARGET_PATH"] = None
-        flags_dict["LOG_PATH"] = self.log_path
-        flags_dict["LOG_FORMAT"] = self.log_format
-        flags_dict["LOG_FORMAT_FILE"] = self.log_format
-        flags_dict["SEND_ANONYMOUS_USAGE_STATS"] = self.send_anonymous_usage_stats
-        flags_dict["PARTIAL_PARSE"] = self.partial_parse
-        flags_dict["STATIC_PARSER"] = self.static_parser
-        flags_dict["PROFILES_DIR"] = self.profiles_dir
-        flags_dict["STORE_FAILURES"] = self.store_failures
-        flags_dict["LOG_LEVEL"] = self.log_level
-        flags_dict["LOG_LEVEL_FILE"] = self.log_level_file
-        flags_dict["LOG_CACHE_EVENTS"] = self.log_cache_events
-        flags_dict["QUIET"] = self.quiet
-        flags_dict["DEBUG"] = self.debug
-        flags_dict["USE_COLORS_FILE"] = self.use_colors_file
-        flags_dict["WRITE_JSON"] = self.write_json
-        flags_dict["INDIRECT_SELECTION"] = getattr(self, "indirect_selection", None)
-        flags_dict["FAIL_FAST"] = self.fail_fast
-        flags_dict["INTROSPECT"] = self.introspect
-
-        flags_ns = argparse.Namespace(**flags_dict)
-        return flags_ns
-
     @property
     def dbt_task(self) -> Type[BaseTask]:
         """Access to the underlying dbt task class."""
@@ -376,30 +301,36 @@ class BaseConfig:
             A tuple with the corresponding subclass of a dbt BaseTask and the
             RuntimeConfig for the task.
         """
-        if not DBT_INSTALLED_LESS_THAN_1_5:
-            local_flags = self.to_flags()
-            flags.set_flags(local_flags)
-
+        flags.set_from_args(self, {})  # type: ignore
         project, profile = self.create_dbt_project_and_profile(extra_targets)
+
+        if not DBT_INSTALLED_GTE_1_9:
+            self.cls.set_log_format()  # type: ignore[attr-defined]
+
+        initialize_from_flags(self.send_anonymous_usage_stats, self.profiles_dir)
+
+        local_flags = flags.get_flags()
+
         runtime_config = self.create_runtime_config(project, profile)
 
-        if not DBT_INSTALLED_LESS_THAN_1_5:
-            if issubclass(self.dbt_task, ConfiguredTask) and runtime_config:
-                register_adapter(runtime_config)
-                manifest = ManifestLoader.get_full_manifest(
-                    runtime_config, write_perf_info=write_perf_info
-                )
-                task: BaseTask = self.dbt_task(
-                    args=self, config=runtime_config, manifest=manifest
-                )
-            elif issubclass(self.dbt_task, DepsTask):
-                task = self.dbt_task(args=self, project=project)
-            elif issubclass(self.dbt_task, DebugTask):
-                task = self.dbt_task(args=self, config=runtime_config)
-            else:
-                task = self.dbt_task(args=self, config=runtime_config, project=project)
+        if issubclass(self.dbt_task, ConfiguredTask) and runtime_config:
+            manifest = parse_manifest(
+                runtime_config,
+                write_perf_info=write_perf_info,
+                write=False,
+                write_json=False,
+            )
+            task: BaseTask = self.dbt_task(
+                args=local_flags, config=runtime_config, manifest=manifest
+            )
+        elif issubclass(self.dbt_task, DepsTask):
+            task = self.dbt_task(args=local_flags, project=project)
+        elif issubclass(self.dbt_task, DebugTask):
+            task = self.dbt_task(args=local_flags)
+        elif issubclass(self.dbt_task, CleanTask):
+            task = self.dbt_task(args=local_flags, config=project)
         else:
-            task = self.dbt_task(args=self, config=runtime_config)
+            task = self.dbt_task(args=local_flags)
 
         if self.compiled_target is not None:
             # Only supported by subclasses of dbt's GraphRunnableTask.
@@ -415,7 +346,6 @@ class BaseConfig:
 
         If the given task's ConfigType is not RuntimeConfig, None will be returned
         instead.
-        Flags are also initialized as configuration is created.
 
         Args:
             project: dbt project.
@@ -424,17 +354,13 @@ class BaseConfig:
         Returns:
             A RuntimeConfig instance.
         """
-        initialize_from_flags(self.send_anonymous_usage_stats, self.profiles_dir)
-        self.cls.set_log_format()
-        flags.set_from_args(self, profile.user_config)  # type: ignore
-
-        cfg_type = self.dbt_task.ConfigType
+        if isinstance(self.vars, str):
+            self.vars = yaml.safe_load(self.vars)
 
-        if issubclass(cfg_type, RuntimeConfig):
-            if not DBT_INSTALLED_LESS_THAN_1_5 and isinstance(self.vars, str):
-                self.vars = yaml.safe_load(self.vars)
-            return cfg_type.from_parts(project=project, profile=profile, args=self)
-        return None
+        try:
+            return RuntimeConfig.from_parts(project=project, profile=profile, args=self)
+        except DbtProjectError:
+            return None
 
     def create_dbt_project_and_profile(
         self, extra_targets: Optional[dict[str, Any]] = None
@@ -449,7 +375,7 @@ class BaseConfig:
         """
         profile = self.create_dbt_profile(extra_targets)
 
-        project_renderer = DbtProjectYamlRenderer(profile, self.parsed_vars)
+        project_renderer = DbtProjectYamlRenderer(profile, self.vars)
         project = Project.from_project_root(
             self.project_dir or ".",
             project_renderer,
@@ -480,7 +406,7 @@ class BaseConfig:
     @property
     def profile_renderer(self) -> ProfileRenderer:
         """Return a ProfileRenderer with this config's parsed vars."""
-        profile_renderer = ProfileRenderer(self.parsed_vars)
+        profile_renderer = ProfileRenderer(self.vars)
         return profile_renderer
 
     @property
@@ -513,6 +439,11 @@ class BaseConfig:
         Returns:
             A dbt profile for the task represented by this configuration.
         """
+        from dbt_common.clients.system import get_env
+        from dbt_common.context import set_invocation_context
+
+        set_invocation_context(get_env())
+
         if self.profiles_dir is not None:
             raw_profiles = read_profile(self.profiles_dir)
         else:
@@ -524,15 +455,12 @@ class BaseConfig:
             outputs.setdefault("target", self.target)
             profile["outputs"] = {**outputs, **extra_targets}
 
-        user_config = raw_profiles.get("config", {})
-
         profile = Profile.from_raw_profile_info(
             raw_profile=raw_profiles.get(
                 self.profile_name, {}
             ),  # Let dbt handle missing profile errors.
             profile_name=self.profile_name,
             renderer=self.profile_renderer,
-            user_config=user_config,
             target_override=self.target,
             threads_override=self.threads,
         )
@@ -668,7 +596,7 @@ class ParseTaskConfig(SelectionConfig):
     TODO: Temporarily, handle this as a CompileTask.
     """
 
-    cls: Type[BaseTask] = dataclasses.field(default=ParseTask, init=False)
+    cls: Type[BaseTask] = dataclasses.field(default=CompileTask, init=False)
     compile: Optional[bool] = None
     which: str = dataclasses.field(default="parse", init=False)
     write_manifest: Optional[bool] = None
@@ -694,8 +622,6 @@ class RunOperationTaskConfig(BaseConfig):
     def __post_init__(self):
         """Support dictionary args by casting them to str after setting."""
         super().__post_init__()
-        if DBT_INSTALLED_LESS_THAN_1_5:
-            self.args = str(self.args)
 
 
 @dataclasses.dataclass
@@ -734,6 +660,7 @@ class TestTaskConfig(SelectionConfig):
     singular: Optional[bool] = None
     store_failures: Optional[bool] = None
     which: str = dataclasses.field(default="test", init=False)
+    resource_types: Optional[list[str]] = None
 
     def __post_init__(self):
         """Support for type casting arguments."""

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/utils/enums.py

@@ -1,4 +1,5 @@
 """Enumeration subclasses used in dbt configurations."""
+
 from enum import Enum
 from typing import Type, TypeVar
 
@@ -24,6 +25,7 @@ class LogFormat(str, Enum):
     """Allowed dbt log formats."""
 
     DEFAULT = "default"
+    DEBUG = "debug"
     JSON = "json"
     TEXT = "text"
 

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/utils/env.py

@@ -1,4 +1,5 @@
 """Provides utilities to interact with environment variables."""
+
 import copy
 import os
 from contextlib import contextmanager

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/airflow_dbt_python/utils/url.py

@@ -1,4 +1,5 @@
 """A URL class to locate dbt resources."""
+
 from __future__ import annotations
 
 import os
@@ -464,7 +465,7 @@ def tar_dir_from_url(url: URL, tar_url: URL) -> None:
     except IndexError:
         compression = ""
 
-    with tarfile.open(tar_url, f"w:{compression}") as tf:
+    with tarfile.open(tar_url, f"w:{compression}") as tf:  # type: ignore[call-overload]
         for _file in url:
             tf.add(_file)
 
@@ -472,6 +473,6 @@ def tar_dir_from_url(url: URL, tar_url: URL) -> None:
 if __name__ == "__main__":
     import doctest
 
-    import airflow_dbt_python.utils.url as url
+    import airflow_dbt_python.utils.url as url_utils
 
-    doctest.testmod(url)
+    doctest.testmod(url_utils)

airflow_dbt_python-2.2.0/airflow_dbt_python/utils/version.py

@@ -0,0 +1,26 @@
+"""Provides utilities to differentiate between installed dbt versions.
+
+These are only used to ensure backwards compatibility with older versions of dbt.
+"""
+
+try:
+    from dbt.semver import Matchers, VersionSpecifier
+except ImportError:
+    from dbt_common.semver import Matchers, VersionSpecifier
+
+from dbt.version import installed
+
+DBT_1_8 = VersionSpecifier(
+    major="1", minor="8", patch="0", matcher=Matchers.GREATER_THAN_OR_EQUAL
+)
+DBT_1_9 = VersionSpecifier(
+    major="1", minor="9", patch="0", matcher=Matchers.GREATER_THAN_OR_EQUAL
+)
+DBT_2_0 = VersionSpecifier(
+    major="2", minor="0", patch="0", matcher=Matchers.GREATER_THAN_OR_EQUAL
+)
+
+DBT_INSTALLED_GTE_1_9 = installed.compare(DBT_1_9) == 1
+
+DBT_INSTALLED_1_8 = DBT_1_8 < installed < DBT_1_9
+DBT_INSTALLED_1_9 = DBT_1_9 < installed < DBT_2_0

{airflow_dbt_python-2.0.1 → airflow_dbt_python-2.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "airflow-dbt-python"
-version = "2.0.1"
+version = "2.2.0"
 description = "A collection of Airflow operators, hooks, and utilities to execute dbt commands"
 authors = ["Tomás Farías Santana <tomas@tomasfarias.dev>"]
 license = "MIT"
@@ -13,46 +13,31 @@ classifiers = [
 
     "License :: OSI Approved :: MIT License",
 
-    "Programming Language :: Python :: 3.8",
     "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
 ]
 
+[tool.poetry.requires-plugins]
+poetry-plugin-export = ">=1.8"
+
 [tool.poetry.dependencies]
-python = ">=3.8,<3.12"  # Airflow providers require this upper boundary
+python = ">=3.9,<3.13"  # Airflow providers require this upper boundary
 
-apache-airflow = { version = ">=2.2" }
+apache-airflow = { version = ">=2.8" }
 apache-airflow-providers-amazon = { version = ">=3.0.0", optional = true }
 apache-airflow-providers-ssh = { version = ">=3.0.0", optional = true }
 connexion = { version = "<3.0" }  # https://github.com/apache/airflow/issues/35217
 dulwich = { version = ">=0.21", optional = true }
 contextlib-chdir = { version = "==1.0.2", python = "<3.11" }
 
-dbt-core = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-bigquery = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-postgres = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-redshift = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-snowflake = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-spark = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
+dbt-core = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-bigquery = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-postgres = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-redshift = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-snowflake = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-spark = { version = ">=1.8.0,<2.0.0", optional = true }
 
 [tool.poetry.group.dev]
 optional = true
@@ -63,15 +48,15 @@ apache-airflow-providers-ssh = ">=3.0.0"
 black = ">=22"
 boto3-stubs = { extras = ["s3"], version = ">=1.26.8" }
 coverage = { extras = ["toml"], version = ">=7.2" }
-dbt-postgres = ">=1.0.0,<1.8.0"
+# docutils 0.21 causes an error with poetry (https://github.com/python-poetry/poetry/issues/9293)
+docutils = "!=0.21"
 dulwich = ">=0.21,!=0.21.6"
 moto = ">=4.0.3"
 mypy = ">=1"
 freezegun = ">=1.1.0"
 pre-commit = ">=2.18"
-psycopg2-binary = ">=2.8.6"
 pytest = ">=7.2"
-pytest-postgresql = ">=3,<4"  # dbt-postgres doesn't support psycopg3 yet
+pytest-postgresql = ">=5"
 ruff = ">=0.0.254"
 types-freezegun = ">=1.1.6"
 types-PyYAML = ">=6.0.7"
@@ -81,38 +66,22 @@ optional = true
 
 [tool.poetry.group.docs.dependencies]
 Sphinx = [
-    { python = "<3.8", version = "<6.0" },
-    { python = ">=3.8,<3.9", version = "<7.2" },
-    { python = ">=3.9", version = ">=7.2" },
+    { python = ">=3.9", version = "7.4.7" },
 ]
-sphinx-rtd-theme = ">=1.1"
-sphinx-copybutton = ">=0.5"
+sphinx-rtd-theme = "3.0.2"
+sphinx-copybutton = "0.5.2"
 attrs = ">=22"
+zipp = ">=3.19.1"
 
 [tool.poetry.group.adapters]
 optional = true
 
 [tool.poetry.group.adapters.dependencies]
-dbt-bigquery = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-postgres = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-redshift = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-snowflake = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
-dbt-spark = [
-    { python = "<3.8", version = ">=1.4.0,<1.6.0", optional = true },
-    { python = ">=3.11", version = ">=1.4.0,<1.8.0", optional = true },
-]
+dbt-bigquery = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-postgres = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-redshift = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-snowflake = { version = ">=1.8.0,<2.0.0", optional = true }
+dbt-spark = { version = ">=1.8.0,<2.0.0", optional = true }
 
 [tool.poetry.extras]
 airflow-providers = ["apache-airflow-providers-amazon", "apache-airflow-providers-ssh"]
@@ -132,6 +101,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.black]
 line-length = 88
+target-version = ["py39", "py310", "py311", "py312"]
 include = '\.pyi?$'
 exclude = '''
 /(
@@ -180,10 +150,8 @@ markers = [
 ]
 
 [tool.ruff]
-select = ["E", "D", "F", "I"]
-ignore = ["D107"]
 line-length = 88
-target-version = "py311"
+target-version = "py39"
 exclude = [
     ".bzr",
     ".direnv",
@@ -209,14 +177,18 @@ exclude = [
     "webserver_config.py",
 ]
 
-[tool.ruff.isort]
+[tool.ruff.lint]
+select = ["E", "D", "F", "I"]
+ignore = ["D107"]
+
+[tool.ruff.lint.isort]
 known-third-party = [
   "airflow",
   "dbt",
   "yaml",
 ]
 
-[tool.ruff.per-file-ignores]
+[tool.ruff.lint.per-file-ignores]
 "__init__.py" = [
     "E402",
     "F401",
@@ -227,5 +199,5 @@ known-third-party = [
 ]
 "docs/*" = ["E402"]
 
-[tool.ruff.pydocstyle]
+[tool.ruff.lint.pydocstyle]
 convention = "google"

airflow_dbt_python-2.0.1/airflow_dbt_python/utils/version.py

@@ -1,30 +0,0 @@
-"""Provides utilities to differentiate between installed dbt versions.
-
-These are only used to ensure backwards compatibility with older versions of dbt.
-"""
-
-from dbt.semver import Matchers, VersionSpecifier
-from dbt.version import installed
-
-DBT_1_5 = VersionSpecifier(
-    major="1", minor="5", patch="0", matcher=Matchers.GREATER_THAN_OR_EQUAL
-)
-DBT_1_6 = VersionSpecifier(
-    major="1", minor="6", patch="0", matcher=Matchers.GREATER_THAN_OR_EQUAL
-)
-DBT_1_7 = VersionSpecifier(
-    major="1", minor="7", patch="0", matcher=Matchers.GREATER_THAN_OR_EQUAL
-)
-DBT_1_8 = VersionSpecifier(
-    major="1", minor="8", patch="0", matcher=Matchers.GREATER_THAN_OR_EQUAL
-)
-
-DBT_INSTALLED_LESS_THAN_1_5 = installed < DBT_1_5
-
-DBT_INSTALLED_GTE_1_5 = installed.compare(DBT_1_5) == 1
-DBT_INSTALLED_GTE_1_6 = installed.compare(DBT_1_6) == 1
-DBT_INSTALLED_GTE_1_7 = installed.compare(DBT_1_7) == 1
-
-DBT_INSTALLED_1_5 = DBT_1_5 < installed < DBT_1_6
-DBT_INSTALLED_1_6 = DBT_1_6 < installed < DBT_1_7
-DBT_INSTALLED_1_7 = DBT_1_7 < installed < DBT_1_8

airflow_dbt_python-2.0.1/setup.py

@@ -1,81 +0,0 @@
-# -*- coding: utf-8 -*-
-from setuptools import setup
-
-packages = \
-['airflow_dbt_python',
- 'airflow_dbt_python.hooks',
- 'airflow_dbt_python.operators',
- 'airflow_dbt_python.utils']
-
-package_data = \
-{'': ['*']}
-
-install_requires = \
-['apache-airflow>=2.2', 'connexion<3.0']
-
-extras_require = \
-{':python_version < "3.11"': ['contextlib-chdir==1.0.2'],
- 'adapters:python_version < "3.8"': ['dbt-bigquery>=1.4.0,<1.6.0',
-                                     'dbt-bigquery>=1.4.0,<1.6.0',
-                                     'dbt-postgres>=1.4.0,<1.6.0',
-                                     'dbt-postgres>=1.4.0,<1.6.0',
-                                     'dbt-redshift>=1.4.0,<1.6.0',
-                                     'dbt-redshift>=1.4.0,<1.6.0',
-                                     'dbt-snowflake>=1.4.0,<1.6.0',
-                                     'dbt-snowflake>=1.4.0,<1.6.0',
-                                     'dbt-spark>=1.4.0,<1.6.0',
-                                     'dbt-spark>=1.4.0,<1.6.0'],
- 'adapters:python_version >= "3.11"': ['dbt-bigquery>=1.4.0,<1.8.0',
-                                       'dbt-bigquery>=1.4.0,<1.8.0',
-                                       'dbt-postgres>=1.4.0,<1.8.0',
-                                       'dbt-postgres>=1.4.0,<1.8.0',
-                                       'dbt-redshift>=1.4.0,<1.8.0',
-                                       'dbt-redshift>=1.4.0,<1.8.0',
-                                       'dbt-snowflake>=1.4.0,<1.8.0',
-                                       'dbt-snowflake>=1.4.0,<1.8.0',
-                                       'dbt-spark>=1.4.0,<1.8.0',
-                                       'dbt-spark>=1.4.0,<1.8.0'],
- 'airflow-providers': ['apache-airflow-providers-amazon>=3.0.0',
-                       'apache-airflow-providers-ssh>=3.0.0'],
- 'bigquery:python_version < "3.8"': ['dbt-bigquery>=1.4.0,<1.6.0',
-                                     'dbt-bigquery>=1.4.0,<1.6.0'],
- 'bigquery:python_version >= "3.11"': ['dbt-bigquery>=1.4.0,<1.8.0',
-                                       'dbt-bigquery>=1.4.0,<1.8.0'],
- 'git': ['apache-airflow-providers-ssh>=3.0.0', 'dulwich>=0.21'],
- 'postgres:python_version < "3.8"': ['dbt-postgres>=1.4.0,<1.6.0',
-                                     'dbt-postgres>=1.4.0,<1.6.0'],
- 'postgres:python_version >= "3.11"': ['dbt-postgres>=1.4.0,<1.8.0',
-                                       'dbt-postgres>=1.4.0,<1.8.0'],
- 'redshift:python_version < "3.8"': ['dbt-redshift>=1.4.0,<1.6.0',
-                                     'dbt-redshift>=1.4.0,<1.6.0'],
- 'redshift:python_version >= "3.11"': ['dbt-redshift>=1.4.0,<1.8.0',
-                                       'dbt-redshift>=1.4.0,<1.8.0'],
- 's3': ['apache-airflow-providers-amazon>=3.0.0'],
- 'snowflake:python_version < "3.8"': ['dbt-snowflake>=1.4.0,<1.6.0',
-                                      'dbt-snowflake>=1.4.0,<1.6.0'],
- 'snowflake:python_version >= "3.11"': ['dbt-snowflake>=1.4.0,<1.8.0',
-                                        'dbt-snowflake>=1.4.0,<1.8.0'],
- 'spark:python_version < "3.8"': ['dbt-spark>=1.4.0,<1.6.0',
-                                  'dbt-spark>=1.4.0,<1.6.0'],
- 'spark:python_version >= "3.11"': ['dbt-spark>=1.4.0,<1.8.0',
-                                    'dbt-spark>=1.4.0,<1.8.0']}
-
-setup_kwargs = {
-    'name': 'airflow-dbt-python',
-    'version': '2.0.1',
-    'description': 'A collection of Airflow operators, hooks, and utilities to execute dbt commands',
-    'long_description': '# airflow-dbt-python\n\n[![PyPI version](https://img.shields.io/pypi/v/airflow-dbt-python?style=plastic)](https://pypi.org/project/airflow-dbt-python/)\n[![CI testing](https://github.com/tomasfarias/airflow-dbt-python/actions/workflows/ci.yaml/badge.svg)](https://github.com/tomasfarias/airflow-dbt-python/actions)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Test coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/tomasfarias/81ef37701aa088d18db8a58ce07c79c7/raw/covbadge.json)](https://github.com/tomasfarias/airflow-dbt-python/actions)\n[![Documentation](https://readthedocs.org/projects/airflow-dbt-python/badge/?version=latest)](https://airflow-dbt-python.readthedocs.io/en/latest/?badge=latest)\n\nA collection of [Airflow](https://airflow.apache.org/) operators, hooks, and utilities to execute [`dbt`](https://pypi.org/project/dbt-core/) commands.\n\nRead the [documentation](https://airflow-dbt-python.readthedocs.io) for examples, installation instructions, and more details.\n\n# Installation\n\n## Requirements\n\nBefore using *airflow-dbt-python*, ensure you meet the following requirements:\n* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.4.0 or later.\n* An Airflow environment using version 2.2 or later.\n\n  * If using any managed service, like AWS MWAA, ensure your environment is created with a supported version of Airflow.\n  * If self-hosting, Airflow installation instructions can be found in their [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).\n\n* Running Python 3.8 or later in your Airflow environment.\n\n> **Warning**\n>\n> Even though we don\'t impose any upper limits on versions of Airflow and *dbt*, it\'s possible that new versions are not supported immediately after release, particularly for *dbt*. We recommend testing the latest versions before upgrading and [reporting any issues](https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose).\n\n> **Note**\n>\n> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/).\n\n## From PyPI\n\n*airflow-dbt-python* is available in [PyPI](https://pypi.org/project/airflow-dbt-python/) and can be installed with *pip*:\n\n``` shell\npip install airflow-dbt-python\n```\n\nAs a convenience, some *dbt* adapters can be installed by specifying extras. For example, if requiring the *dbt-redshift* adapter:\n\n``` shell\npip install airflow-dbt-python[redshift]\n```\n\n## From this repo\n\n*airflow-dbt-python* can also be built from source by cloning this GitHub repository:\n\n``` shell\ngit clone https://github.com/tomasfarias/airflow-dbt-python.git\ncd airflow-dbt-python\n```\n\nAnd installing with *Poetry*:\n\n``` shell\npoetry install\n```\n\n## In AWS MWAA\n\nAdd *airflow-dbt-python* to your `requirements.txt` file and edit your Airflow environment to use this new `requirements.txt` file, or upload it as a plugin.\n\nRead the [documentation](https://airflow-dbt-python.readthedocs.io/en/latest/getting_started.html#installing-in-mwaa) for more a more detailed AWS MWAA installation breakdown.\n\n## In other managed services\n\n*airflow-dbt-python* should be compatible with most or all Airflow managed services. Consult the documentation specific to your provider.\n\nIf you notice an issue when installing *airflow-dbt-python* in a specific managed service, please open an [issue](https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose).\n\n# Features\n\n*airflow-dbt-python* aims to make dbt a **first-class citizen** of Airflow by supporting additional features that integrate both tools. As you would expect, *airflow-dbt-python* can run all your dbt workflows in Airflow with the same interface you are used to from the CLI, but without being a mere wrapper: *airflow-dbt-python* directly communicates with internal *dbt-core* classes, bridging the gap between them and Airflow\'s operator interface. Essentially, we are attempting to use *dbt* **as a library**.\n\nAs this integration was completed, several features were developed to **extend the capabilities of dbt** to leverage Airflow as much as possible. Can you think of a way *dbt* could leverage Airflow that is not currently supported? Let us know in a [GitHub issue](https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose)!\n\n## Independent task execution\n\nAirflow executes [Tasks](https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html) independent of one another: even though downstream and upstream dependencies between tasks exist, the execution of an individual task happens entirely independently of any other task execution (see: [Tasks Relationships](https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html#relationships)).\n\nIn order to work with this constraint, *airflow-dbt-python* runs each dbt command in a **temporary and isolated directory**. Before execution, all the relevant dbt files are copied from supported backends, and after executing the command any artifacts are exported. This ensures dbt can work with any Airflow deployment, including most production deployments as they are usually running [Remote Executors](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html#executor-types) and do not guarantee any files will be shared by default between tasks, since each task may run in a completely different environment.\n\n\n## Download dbt files from a remote storage\n\nThe dbt parameters `profiles_dir` and `project_dir` would normally point to a directory containing a `profiles.yml` file and a dbt project in the local environment respectively (defined by the presence of a *dbt_project.yml* file). *airflow-dbt-python* extends these parameters to also accept an URL pointing to a remote storage.\n\nCurrently, we support the following remote storages:\n\n* [AWS S3](https://aws.amazon.com/s3/) (identified by a *s3* scheme).\n* Remote git repositories, like those stored in GitHub (both *https* and *ssh* schemes are supported).\n\n* If a remote URL is used for `project_dir`, then this URL must point to a location in your remote storage containing a *dbt* project to run. A *dbt* project is identified by the prescence of a *dbt_project.yml*, and contains all your [resources](https://docs.getdbt.com/docs/build/projects). All of the contents of this remote location will be downloaded and made available for the operator. The URL may also point to an archived file containing all the files of a dbt project, which will be downloaded, uncompressed, and made available for the operator.\n* If a remote URL is used for `profiles_dir`, then this URL must point to a location in your remote storage that contains a *profiles.yml* file. The *profiles.yml* file will be downloaded and made available for the operator to use when running. The *profiles.yml* may be part of your *dbt* project, in which case this argument may be ommitted.\n\nThis feature is intended to work in line with Airflow\'s [description of the task concept](https://airflow.apache.org/docs/apache-airflow/stable/concepts/tasks.html#relationships):\n\n> Tasks don’t pass information to each other by default, and run entirely independently.\n\nWe interpret this as meaning a task should be responsible of fetching all the *dbt* related files it needs in order to run independently, as already described in [Independent Task Execution](#independent-task-execution).\n\n## Push dbt artifacts to XCom\n\nEach dbt execution produces one or more [JSON artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts/) that are valuable to produce meta-metrics, build conditional workflows, for reporting purposes, and other uses. *airflow-dbt-python* can push these artifacts to [XCom](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html) as requested via the `do_xcom_push_artifacts` parameter, which takes a list of artifacts to push.\n\n## Use Airflow connections as dbt targets (without a profiles.yml)\n\n[Airflow connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html) allow users to manage and store connection information, such as hostname, port, username, and password, for operators to use when accessing certain applications, like databases. Similarly, a *dbt* `profiles.yml` file stores connection information under each target key. *airflow-dbt-python* bridges the gap between the two and allows you to use connection information stored as an Airflow connection by specifying the connection id as the `target` parameter of any of the *dbt* operators it provides. What\'s more, if using an Airflow connection, the `profiles.yml` file may be entirely omitted (although keep in mind a `profiles.yml` file contains a configuration block besides target connection information).\n\nSee an example DAG [here](examples/airflow_connection_target_dag.py).\n\n# Motivation\n\n## Airflow running in a managed environment\n\nAlthough [`dbt`](https://docs.getdbt.com/) is meant to be installed and used as a CLI, we may not have control of the environment where Airflow is running, disallowing us the option of using *dbt* as a CLI.\n\nThis is exactly what happens when using [Amazon\'s Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) or MWAA: although a list of Python requirements can be passed, the CLI cannot be found in the worker\'s PATH.\n\nThere is a workaround which involves using Airflow\'s `BashOperator` and running Python from the command line:\n\n``` python\nfrom airflow.operators.bash import BashOperator\n\nBASH_COMMAND = "python -c \'from dbt.main import main; main()\' run"\noperator = BashOperator(\n    task_id="dbt_run",\n    bash_command=BASH_COMMAND,\n)\n```\n\nBut it can get cumbersome when appending all potential arguments a `dbt run` command (or other subcommand) can take.\n\nThat\'s where *airflow-dbt-python* comes in: it abstracts the complexity of interfacing with *dbt-core* and exposes one operator for each *dbt* subcommand that can be instantiated with all the corresponding arguments that the *dbt* CLI would take.\n\n## An alternative to *airflow-dbt* that works without the *dbt* CLI\n\nThe alternative [`airflow-dbt`](https://pypi.org/project/airflow-dbt/) package, by default, would not work if the *dbt* CLI is not in PATH, which means it would not be usable in MWAA. There is a workaround via the `dbt_bin` argument, which can be set to `"python -c \'from dbt.main import main; main()\' run"`, in similar fashion as the `BashOperator` example. Yet this approach is not without its limitations:\n* *airflow-dbt* works by wrapping the *dbt* CLI, which makes our code dependent on the environment in which it runs.\n* *airflow-dbt* does not support the full range of arguments a command can take. For example, `DbtRunOperator` does not have an attribute for `fail_fast`.\n* *airflow-dbt* does not offer access to *dbt* artifacts created during execution. *airflow-dbt-python* does so by pushing any artifacts to [XCom](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html).\n\n# Usage\n\nCurrently, the following *dbt* commands are supported:\n\n* `clean`\n* `compile`\n* `debug`\n* `deps`\n* `docs generate`\n* `ls`\n* `parse`\n* `run`\n* `run-operation`\n* `seed`\n* `snapshot`\n* `source`\n* `test`\n\n## Examples\n\nAll example DAGs are tested against the latest Airflow version. Some changes, like modifying `import` statements or changing types, may be required for them to work in other versions.\n\n``` python\nimport datetime as dt\n\nimport pendulum\nfrom airflow import DAG\n\nfrom airflow_dbt_python.operators.dbt import (\n    DbtRunOperator,\n    DbtSeedOperator,\n    DbtTestOperator,\n)\n\nargs = {\n    "owner": "airflow",\n}\n\nwith DAG(\n    dag_id="example_dbt_operator",\n    default_args=args,\n    schedule="0 0 * * *",\n    start_date=pendulum.today("UTC").add(days=-1),\n    dagrun_timeout=dt.timedelta(minutes=60),\n    tags=["example", "example2"],\n) as dag:\n    dbt_test = DbtTestOperator(\n        task_id="dbt_test",\n        selector_name="pre-run-tests",\n    )\n\n    dbt_seed = DbtSeedOperator(\n        task_id="dbt_seed",\n        select=["/path/to/first.csv", "/path/to/second.csv"],\n        full_refresh=True,\n    )\n\n    dbt_run = DbtRunOperator(\n        task_id="dbt_run",\n        select=["/path/to/models"],\n        full_refresh=True,\n        fail_fast=True,\n    )\n\n    dbt_test >> dbt_seed >> dbt_run\n```\n\nMore examples can be found in the [`examples/`](examples/) directory and the [documentation](https://airflow-dbt-python.readthedocs.io).\n\n# Development\n\nSee the [development documentation](https://airflow-dbt-python.readthedocs.io/en/latest/development.html) for a more in-depth dive into setting up a development environment, running the test-suite, and general commentary on working on *airflow-dbt-python*.\n\n## Testing\n\nTests are run with *pytest*, can be located in `tests/`. To run them locally, you may use *Poetry*:\n\n``` shell\npoetry run pytest tests/ -vv\n```\n\n# License\n\nThis project is licensed under the MIT license. See [LICENSE](LICENSE).\n',
-    'author': 'Tomás Farías Santana',
-    'author_email': 'tomas@tomasfarias.dev',
-    'maintainer': 'None',
-    'maintainer_email': 'None',
-    'url': 'https://github.com/tomasfarias/airflow-dbt-python',
-    'packages': packages,
-    'package_data': package_data,
-    'install_requires': install_requires,
-    'extras_require': extras_require,
-    'python_requires': '>=3.8,<3.12',
-}
-
-
-setup(**setup_kwargs)