apache-airflow-providers-common-sql 1.23.0__tar.gz → 1.24.0__tar.gz

{apache_airflow_providers_common_sql-1.23.0 → apache_airflow_providers_common_sql-1.24.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: apache-airflow-providers-common-sql
-Version: 1.23.0
+Version: 1.24.0
 Summary: Provider package apache-airflow-providers-common-sql for Apache Airflow
 Keywords: airflow-provider,common.sql,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -23,11 +23,12 @@ Classifier: Topic :: System :: Monitoring
 Requires-Dist: apache-airflow>=2.9.0
 Requires-Dist: sqlparse>=0.5.1
 Requires-Dist: more-itertools>=9.0.0
+Requires-Dist: methodtools>=0.4.7
 Requires-Dist: apache-airflow-providers-openlineage ; extra == "openlineage"
 Requires-Dist: pandas>=2.1.2,<2.2 ; extra == "pandas"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
 Project-URL: Twitter, https://x.com/ApacheAirflow
@@ -36,32 +37,31 @@ Provides-Extra: openlineage
 Provides-Extra: pandas
 
 
- .. Licensed to the Apache Software Foundation (ASF) under one
-    or more contributor license agreements.  See the NOTICE file
-    distributed with this work for additional information
-    regarding copyright ownership.  The ASF licenses this file
-    to you under the Apache License, Version 2.0 (the
-    "License"); you may not use this file except in compliance
-    with the License.  You may obtain a copy of the License at
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
 
- ..   http://www.apache.org/licenses/LICENSE-2.0
+..   http://www.apache.org/licenses/LICENSE-2.0
 
- .. Unless required by applicable law or agreed to in writing,
-    software distributed under the License is distributed on an
-    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-    KIND, either express or implied.  See the License for the
-    specific language governing permissions and limitations
-    under the License.
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
 
- .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
-
- .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
-    `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+.. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
+.. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+   ``PROVIDER_README_TEMPLATE.rst.jinja2`` IN the ``dev/breeze/src/airflow_breeze/templates`` DIRECTORY
 
 Package ``apache-airflow-providers-common-sql``
 
-Release: ``1.23.0``
+Release: ``1.24.0``
 
 
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
@@ -74,7 +74,7 @@ This is a provider package for ``common.sql`` provider. All classes for this pro
 are in ``airflow.providers.common.sql`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0/>`_.
 
 Installation
 ------------
@@ -94,6 +94,7 @@ PIP package         Version required
 ``apache-airflow``  ``>=2.9.0``
 ``sqlparse``        ``>=0.5.1``
 ``more-itertools``  ``>=9.0.0``
+``methodtools``     ``>=0.4.7``
 ==================  ==================
 
 Cross provider package dependencies
@@ -116,5 +117,5 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0/changelog.html>`_.
 

{apache_airflow_providers_common_sql-1.23.0 → apache_airflow_providers_common_sql-1.24.0}/README.rst

@@ -1,30 +1,29 @@
 
- .. Licensed to the Apache Software Foundation (ASF) under one
-    or more contributor license agreements.  See the NOTICE file
-    distributed with this work for additional information
-    regarding copyright ownership.  The ASF licenses this file
-    to you under the Apache License, Version 2.0 (the
-    "License"); you may not use this file except in compliance
-    with the License.  You may obtain a copy of the License at
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
 
- ..   http://www.apache.org/licenses/LICENSE-2.0
+..   http://www.apache.org/licenses/LICENSE-2.0
 
- .. Unless required by applicable law or agreed to in writing,
-    software distributed under the License is distributed on an
-    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-    KIND, either express or implied.  See the License for the
-    specific language governing permissions and limitations
-    under the License.
+.. Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
 
- .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
-
- .. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
-    `PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+.. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
 
+.. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+   ``PROVIDER_README_TEMPLATE.rst.jinja2`` IN the ``dev/breeze/src/airflow_breeze/templates`` DIRECTORY
 
 Package ``apache-airflow-providers-common-sql``
 
-Release: ``1.23.0``
+Release: ``1.24.0``
 
 
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
@@ -37,7 +36,7 @@ This is a provider package for ``common.sql`` provider. All classes for this pro
 are in ``airflow.providers.common.sql`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0/>`_.
 
 Installation
 ------------
@@ -57,6 +56,7 @@ PIP package         Version required
 ``apache-airflow``  ``>=2.9.0``
 ``sqlparse``        ``>=0.5.1``
 ``more-itertools``  ``>=9.0.0``
+``methodtools``     ``>=0.4.7``
 ==================  ==================
 
 Cross provider package dependencies
@@ -79,4 +79,4 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0/changelog.html>`_.

{apache_airflow_providers_common_sql-1.23.0 → apache_airflow_providers_common_sql-1.24.0}/pyproject.toml

@@ -20,12 +20,12 @@
 # IF YOU WANT TO MODIFY THIS FILE EXCEPT DEPENDENCIES, YOU SHOULD MODIFY THE TEMPLATE
 # `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
 [build-system]
-requires = ["flit_core==3.10.1"]
+requires = ["flit_core==3.11.0"]
 build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-common-sql"
-version = "1.23.0"
+version = "1.24.0"
 description = "Provider package apache-airflow-providers-common-sql for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -52,12 +52,17 @@ classifiers = [
 ]
 requires-python = "~=3.9"
 
-# The dependencies should be modified in place in the generated file
+# The dependencies should be modified in place in the generated file.
 # Any change in the dependencies is preserved when the file is regenerated
+# Make sure to run ``breeze static-checks --type update-providers-dependencies --all-files``
+# After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
 dependencies = [
     "apache-airflow>=2.9.0",
     "sqlparse>=0.5.1",
     "more-itertools>=9.0.0",
+    # The methodtools dependency is necessary since the introduction of dialects:
+    # https://github.com/apache/airflow/pull/41327/files
+    "methodtools>=0.4.7"
 ]
 
 # The optional dependencies should be modified in place in the generated file
@@ -74,9 +79,28 @@ dependencies = [
     "apache-airflow-providers-openlineage"
 ]
 
+[dependency-groups]
+dev = [
+    "apache-airflow",
+    "apache-airflow-task-sdk",
+    "apache-airflow-devel-common",
+    "apache-airflow-providers-openlineage",
+    # Additional devel dependencies (do not remove this line and add extra development dependencies)
+]
+
+[tool.uv.sources]
+# These names must match the names as defined in the pyproject.toml of the workspace items,
+# *not* the workspace folder paths
+apache-airflow = {workspace = true}
+apache-airflow-devel-common = {workspace = true}
+apache-airflow-task-sdk = {workspace = true}
+apache-airflow-providers-common-sql = {workspace = true}
+apache-airflow-providers-fab = {workspace = true}
+apache-airflow-providers-standard = {workspace = true}
+
 [project.urls]
-"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0"
-"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.23.0/changelog.html"
+"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0"
+"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.24.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"
@@ -88,6 +112,3 @@ provider_info = "airflow.providers.common.sql.get_provider_info:get_provider_inf
 
 [tool.flit.module]
 name = "airflow.providers.common.sql"
-
-[tool.pytest.ini_options]
-ignore = "tests/system/"

{apache_airflow_providers_common_sql-1.23.0 → apache_airflow_providers_common_sql-1.24.0}/src/airflow/providers/common/sql/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "1.23.0"
+__version__ = "1.24.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.9.0"

{apache_airflow_providers_common_sql-1.23.0 → apache_airflow_providers_common_sql-1.24.0}/src/airflow/providers/common/sql/get_provider_info.py

@@ -27,8 +27,9 @@ def get_provider_info():
         "name": "Common SQL",
         "description": "`Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__\n",
         "state": "ready",
-        "source-date-epoch": 1739959337,
+        "source-date-epoch": 1741508670,
         "versions": [
+            "1.24.0",
             "1.23.0",
             "1.21.0",
             "1.20.0",
@@ -81,7 +82,10 @@ def get_provider_info():
         "operators": [
             {
                 "integration-name": "Common SQL",
-                "python-modules": ["airflow.providers.common.sql.operators.sql"],
+                "python-modules": [
+                    "airflow.providers.common.sql.operators.sql",
+                    "airflow.providers.common.sql.operators.generic_transfer",
+                ],
             }
         ],
         "dialects": [
@@ -99,12 +103,24 @@ def get_provider_info():
                 ],
             }
         ],
+        "triggers": [
+            {
+                "integration-name": "Common SQL",
+                "python-modules": ["airflow.providers.common.sql.triggers.sql"],
+            }
+        ],
         "sensors": [
             {"integration-name": "Common SQL", "python-modules": ["airflow.providers.common.sql.sensors.sql"]}
         ],
-        "dependencies": ["apache-airflow>=2.9.0", "sqlparse>=0.5.1", "more-itertools>=9.0.0"],
+        "dependencies": [
+            "apache-airflow>=2.9.0",
+            "sqlparse>=0.5.1",
+            "more-itertools>=9.0.0",
+            "methodtools>=0.4.7",
+        ],
         "optional-dependencies": {
             "pandas": ["pandas>=2.1.2,<2.2"],
             "openlineage": ["apache-airflow-providers-openlineage"],
         },
+        "devel-dependencies": [],
     }

apache_airflow_providers_common_sql-1.24.0/src/airflow/providers/common/sql/operators/generic_transfer.py

@@ -0,0 +1,219 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from collections.abc import Sequence
+from functools import cached_property
+from typing import TYPE_CHECKING, Any
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.models import BaseOperator
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+from airflow.providers.common.sql.triggers.sql import SQLExecuteQueryTrigger
+
+if TYPE_CHECKING:
+    import jinja2
+
+    try:
+        from airflow.sdk.definitions.context import Context
+    except ImportError:
+        # TODO: Remove once provider drops support for Airflow 2
+        from airflow.utils.context import Context
+
+
+class GenericTransfer(BaseOperator):
+    """
+    Moves data from a connection to another.
+
+    Assuming that they both provide the required methods in their respective hooks.
+    The source hook needs to expose a `get_records` method, and the destination a
+    `insert_rows` method.
+
+    This is meant to be used on small-ish datasets that fit in memory.
+
+    :param sql: SQL query to execute against the source database. (templated)
+    :param destination_table: target table. (templated)
+    :param source_conn_id: source connection. (templated)
+    :param source_hook_params: source hook parameters.
+    :param destination_conn_id: destination connection. (templated)
+    :param destination_hook_params: destination hook parameters.
+    :param preoperator: sql statement or list of statements to be
+        executed prior to loading the data. (templated)
+    :param insert_args: extra params for `insert_rows` method.
+    :param page_size: number of records to be read in paginated mode (optional).
+    """
+
+    template_fields: Sequence[str] = (
+        "source_conn_id",
+        "destination_conn_id",
+        "sql",
+        "destination_table",
+        "preoperator",
+        "insert_args",
+    )
+    template_ext: Sequence[str] = (
+        ".sql",
+        ".hql",
+    )
+    template_fields_renderers = {"preoperator": "sql"}
+    ui_color = "#b0f07c"
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        destination_table: str,
+        source_conn_id: str,
+        source_hook_params: dict | None = None,
+        destination_conn_id: str,
+        destination_hook_params: dict | None = None,
+        preoperator: str | list[str] | None = None,
+        insert_args: dict | None = None,
+        page_size: int | None = None,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.sql = sql
+        self.destination_table = destination_table
+        self.source_conn_id = source_conn_id
+        self.source_hook_params = source_hook_params
+        self.destination_conn_id = destination_conn_id
+        self.destination_hook_params = destination_hook_params
+        self.preoperator = preoperator
+        self.insert_args = insert_args or {}
+        self.page_size = page_size
+        self._paginated_sql_statement_format = kwargs.get(
+            "paginated_sql_statement_format", "{} LIMIT {} OFFSET {}"
+        )
+
+    @classmethod
+    def get_hook(cls, conn_id: str, hook_params: dict | None = None) -> DbApiHook:
+        """
+        Return DbApiHook for this connection id.
+
+        :param conn_id: connection id
+        :param hook_params: hook parameters
+        :return: DbApiHook for this connection
+        """
+        connection = BaseHook.get_connection(conn_id)
+        hook = connection.get_hook(hook_params=hook_params)
+        if not isinstance(hook, DbApiHook):
+            raise RuntimeError(f"Hook for connection {conn_id!r} must be of type {DbApiHook.__name__}")
+        return hook
+
+    @cached_property
+    def source_hook(self) -> DbApiHook:
+        return self.get_hook(conn_id=self.source_conn_id, hook_params=self.source_hook_params)
+
+    @cached_property
+    def destination_hook(self) -> DbApiHook:
+        return self.get_hook(conn_id=self.destination_conn_id, hook_params=self.destination_hook_params)
+
+    def get_paginated_sql(self, offset: int) -> str:
+        """Format the paginated SQL statement using the current format."""
+        return self._paginated_sql_statement_format.format(self.sql, self.page_size, offset)
+
+    def render_template_fields(
+        self,
+        context: Context,
+        jinja_env: jinja2.Environment | None = None,
+    ) -> None:
+        super().render_template_fields(context=context, jinja_env=jinja_env)
+
+        # Make sure string are converted to integers
+        if isinstance(self.page_size, str):
+            self.page_size = int(self.page_size)
+        commit_every = self.insert_args.get("commit_every")
+        if isinstance(commit_every, str):
+            self.insert_args["commit_every"] = int(commit_every)
+
+    def execute(self, context: Context):
+        if self.preoperator:
+            self.log.info("Running preoperator")
+            self.log.info(self.preoperator)
+            self.destination_hook.run(self.preoperator)
+
+        if self.page_size and isinstance(self.sql, str):
+            self.defer(
+                trigger=SQLExecuteQueryTrigger(
+                    conn_id=self.source_conn_id,
+                    hook_params=self.source_hook_params,
+                    sql=self.get_paginated_sql(0),
+                ),
+                method_name=self.execute_complete.__name__,
+            )
+        else:
+            self.log.info("Extracting data from %s", self.source_conn_id)
+            self.log.info("Executing: \n %s", self.sql)
+
+            results = self.destination_hook.get_records(self.sql)
+
+            self.log.info("Inserting rows into %s", self.destination_conn_id)
+            self.destination_hook.insert_rows(table=self.destination_table, rows=results, **self.insert_args)
+
+    def execute_complete(
+        self,
+        context: Context,
+        event: dict[Any, Any] | None = None,
+    ) -> Any:
+        if event:
+            if event.get("status") == "failure":
+                raise AirflowException(event.get("message"))
+
+            results = event.get("results")
+
+            if results:
+                map_index = context["ti"].map_index
+                offset = (
+                    context["ti"].xcom_pull(
+                        key="offset",
+                        task_ids=self.task_id,
+                        dag_id=self.dag_id,
+                        map_indexes=map_index,
+                        default=0,
+                    )
+                    + self.page_size
+                )
+
+                self.log.info("Offset increased to %d", offset)
+                self.xcom_push(context=context, key="offset", value=offset)
+
+                self.log.info("Inserting %d rows into %s", len(results), self.destination_conn_id)
+                self.destination_hook.insert_rows(
+                    table=self.destination_table, rows=results, **self.insert_args
+                )
+                self.log.info(
+                    "Inserting %d rows into %s done!",
+                    len(results),
+                    self.destination_conn_id,
+                )
+
+                self.defer(
+                    trigger=SQLExecuteQueryTrigger(
+                        conn_id=self.source_conn_id,
+                        hook_params=self.source_hook_params,
+                        sql=self.get_paginated_sql(offset),
+                    ),
+                    method_name=self.execute_complete.__name__,
+                )
+            else:
+                self.log.info(
+                    "No more rows to fetch into %s; ending transfer.",
+                    self.destination_table,
+                )

apache_airflow_providers_common_sql-1.24.0/src/airflow/providers/common/sql/operators/generic_transfer.pyi

@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# This is automatically generated stub for the `common.sql` provider
+#
+# This file is generated automatically by the `update-common-sql-api stubs` pre-commit
+# and the .pyi file represents part of the "public" API that the
+# `common.sql` provider exposes to other providers.
+#
+# Any, potentially breaking change in the stubs will require deliberate manual action from the contributor
+# making a change to the `common.sql` provider. Those stubs are also used by MyPy automatically when checking
+# if only public API of the common.sql provider is used by all the other providers.
+#
+# You can read more in the README_API.md file
+#
+"""
+Definition of the public interface for airflow.providers.common.sql.operators.generic_transfer
+isort:skip_file
+"""
+
+from collections.abc import Sequence
+from functools import cached_property as cached_property
+from typing import Any, ClassVar
+
+import jinja2
+from _typeshed import Incomplete as Incomplete
+
+from airflow.models import BaseOperator
+from airflow.providers.common.sql.hooks.sql import DbApiHook as DbApiHook
+from airflow.utils.context import Context as Context
+
+class GenericTransfer(BaseOperator):
+    template_fields: Sequence[str]
+    template_ext: Sequence[str]
+    template_fields_renderers: ClassVar[dict]
+    ui_color: str
+    sql: Incomplete
+    destination_table: Incomplete
+    source_conn_id: Incomplete
+    source_hook_params: Incomplete
+    destination_conn_id: Incomplete
+    destination_hook_params: Incomplete
+    preoperator: Incomplete
+    insert_args: Incomplete
+    page_size: Incomplete
+    def __init__(
+        self,
+        *,
+        sql: str,
+        destination_table: str,
+        source_conn_id: str,
+        source_hook_params: dict | None = None,
+        destination_conn_id: str,
+        destination_hook_params: dict | None = None,
+        preoperator: str | list[str] | None = None,
+        insert_args: dict | None = None,
+        page_size: int | None = None,
+        **kwargs,
+    ) -> None: ...
+    @classmethod
+    def get_hook(cls, conn_id: str, hook_params: dict | None = None) -> DbApiHook: ...
+    @cached_property
+    def source_hook(self) -> DbApiHook: ...
+    @cached_property
+    def destination_hook(self) -> DbApiHook: ...
+    def get_paginated_sql(self, offset: int) -> str: ...
+    def render_template_fields(
+        self, context: Context, jinja_env: jinja2.Environment | None = None
+    ) -> None: ...
+    def execute(self, context: Context): ...
+    def execute_complete(self, context: Context, event: dict[Any, Any] | None = None) -> Any: ...

{apache_airflow_providers_common_sql-1.23.0 → apache_airflow_providers_common_sql-1.24.0}/src/airflow/providers/common/sql/operators/sql.py

@@ -26,7 +26,8 @@ from typing import TYPE_CHECKING, Any, Callable, ClassVar, NoReturn, SupportsAbs
 from airflow.exceptions import AirflowException, AirflowFailException
 from airflow.hooks.base import BaseHook
 from airflow.models import BaseOperator, SkipMixin
-from airflow.providers.common.sql.hooks.sql import DbApiHook, fetch_all_handler, return_single_query_results
+from airflow.providers.common.sql.hooks.handlers import fetch_all_handler, return_single_query_results
+from airflow.providers.common.sql.hooks.sql import DbApiHook
 from airflow.utils.helpers import merge_dicts
 
 if TYPE_CHECKING:

apache_airflow_providers_common_sql-1.24.0/src/airflow/providers/common/sql/triggers/__init__.py

@@ -0,0 +1,16 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.

apache_airflow_providers_common_sql-1.24.0/src/airflow/providers/common/sql/triggers/sql.py

@@ -0,0 +1,87 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.providers.common.sql.hooks.sql import DbApiHook
+from airflow.triggers.base import BaseTrigger, TriggerEvent
+
+if TYPE_CHECKING:
+    from collections.abc import AsyncIterator
+    from typing import Any
+
+
+class SQLExecuteQueryTrigger(BaseTrigger):
+    """
+    A trigger that executes SQL code in async mode.
+
+    :param sql: the sql statement to be executed (str) or a list of sql statements to execute
+    :param conn_id: the connection ID used to connect to the database
+    :param hook_params: hook parameters
+    """
+
+    def __init__(
+        self,
+        sql: str | list[str],
+        conn_id: str,
+        hook_params: dict | None = None,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.sql = sql
+        self.conn_id = conn_id
+        self.hook_params = hook_params
+
+    def serialize(self) -> tuple[str, dict[str, Any]]:
+        """Serialize the SQLExecuteQueryTrigger arguments and classpath."""
+        return (
+            f"{self.__class__.__module__}.{self.__class__.__name__}",
+            {
+                "sql": self.sql,
+                "conn_id": self.conn_id,
+                "hook_params": self.hook_params,
+            },
+        )
+
+    async def run(self) -> AsyncIterator[TriggerEvent]:
+        try:
+            hook = BaseHook.get_hook(self.conn_id, hook_params=self.hook_params)
+
+            if not isinstance(hook, DbApiHook):
+                raise AirflowException(
+                    f"You are trying to use `common-sql` with {hook.__class__.__name__},"
+                    " but its provider does not support it. Please upgrade the provider to a version that"
+                    " supports `common-sql`. The hook class should be a subclass of"
+                    f" `{hook.__class__.__module__}.{hook.__class__.__name__}`."
+                    f" Got {hook.__class__.__name__} hook with class hierarchy: {hook.__class__.mro()}"
+                )
+
+            self.log.info("Extracting data from %s", self.conn_id)
+            self.log.info("Executing: \n %s", self.sql)
+            self.log.info("Reading records from %s", self.conn_id)
+            results = hook.get_records(self.sql)
+            self.log.info("Reading records from %s done!", self.conn_id)
+
+            self.log.debug("results: %s", results)
+            yield TriggerEvent({"status": "success", "results": results})
+        except Exception as e:
+            self.log.exception("An error occurred: %s", e)
+            yield TriggerEvent({"status": "failure", "message": str(e)})

apache_airflow_providers_common_sql-1.24.0/src/airflow/providers/common/sql/triggers/sql.pyi

@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# This is automatically generated stub for the `common.sql` provider
+#
+# This file is generated automatically by the `update-common-sql-api stubs` pre-commit
+# and the .pyi file represents part of the "public" API that the
+# `common.sql` provider exposes to other providers.
+#
+# Any, potentially breaking change in the stubs will require deliberate manual action from the contributor
+# making a change to the `common.sql` provider. Those stubs are also used by MyPy automatically when checking
+# if only public API of the common.sql provider is used by all the other providers.
+#
+# You can read more in the README_API.md file
+#
+"""
+Definition of the public interface for airflow.providers.common.sql.triggers.sql
+isort:skip_file
+"""
+
+from collections.abc import AsyncIterator
+from typing import Any
+
+from airflow.triggers.base import BaseTrigger as BaseTrigger, TriggerEvent as TriggerEvent
+
+class SQLExecuteQueryTrigger(BaseTrigger):
+    def __init__(
+        self, sql: str | list[str], conn_id: str, hook_params: dict | None = None, **kwargs
+    ) -> None: ...
+    def serialize(self) -> tuple[str, dict[str, Any]]: ...
+    async def run(self) -> AsyncIterator[TriggerEvent]: ...  # type: ignore