apache-airflow-providers-common-sql 1.26.0rc1__tar.gz → 1.27.0rc1__tar.gz

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-common-sql
-Version: 1.26.0rc1
+Version: 1.27.0rc1
 Summary: Provider package apache-airflow-providers-common-sql for Apache Airflow
 Keywords: airflow-provider,common.sql,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -20,7 +20,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
-Requires-Dist: apache-airflow>=2.9.0rc0
+Requires-Dist: apache-airflow>=2.10.0rc1
 Requires-Dist: sqlparse>=0.5.1
 Requires-Dist: more-itertools>=9.0.0
 Requires-Dist: methodtools>=0.4.7
@@ -28,8 +28,8 @@ Requires-Dist: apache-airflow-providers-openlineage ; extra == "openlineage"
 Requires-Dist: pandas>=2.1.2,<2.2 ; extra == "pandas"
 Requires-Dist: polars>=1.26.0 ; extra == "polars"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -63,7 +63,7 @@ Provides-Extra: polars
 
 Package ``apache-airflow-providers-common-sql``
 
-Release: ``1.26.0``
+Release: ``1.27.0``
 
 
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
@@ -76,7 +76,7 @@ This is a provider package for ``common.sql`` provider. All classes for this pro
 are in ``airflow.providers.common.sql`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0/>`_.
 
 Installation
 ------------
@@ -93,7 +93,7 @@ Requirements
 ==================  ==================
 PIP package         Version required
 ==================  ==================
-``apache-airflow``  ``>=2.9.0``
+``apache-airflow``  ``>=2.10.0``
 ``sqlparse``        ``>=0.5.1``
 ``more-itertools``  ``>=9.0.0``
 ``methodtools``     ``>=0.4.7``
@@ -119,5 +119,5 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0/changelog.html>`_.
 

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/README.rst

@@ -23,7 +23,7 @@
 
 Package ``apache-airflow-providers-common-sql``
 
-Release: ``1.26.0``
+Release: ``1.27.0``
 
 
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
@@ -36,7 +36,7 @@ This is a provider package for ``common.sql`` provider. All classes for this pro
 are in ``airflow.providers.common.sql`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0/>`_.
 
 Installation
 ------------
@@ -53,7 +53,7 @@ Requirements
 ==================  ==================
 PIP package         Version required
 ==================  ==================
-``apache-airflow``  ``>=2.9.0``
+``apache-airflow``  ``>=2.10.0``
 ``sqlparse``        ``>=0.5.1``
 ``more-itertools``  ``>=9.0.0``
 ``methodtools``     ``>=0.4.7``
@@ -79,4 +79,4 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0/changelog.html>`_.

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/docs/changelog.rst

@@ -25,6 +25,35 @@
 Changelog
 ---------
 
+1.27.0
+......
+
+.. note::
+  This release of provider is only available for Airflow 2.10+ as explained in the
+  `Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>`_.
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix SADeprecationWarning when using inspector with SQLAlchemy in DbApiHook (#48938)``
+* ``Fix: SQLExecuteQueryOperator does not pass extra_dejson values to hook_params (#49282)``
+
+Misc
+~~~~
+
+* ``Refine type overload for 'common-sql' (#50161)``
+* ``Bump min Airflow version in providers to 2.10 (#49843)``
+* ``refactor: Log generated SQL-statement and passed parameters as ERROR message when an exception occurs during insert_rows (#48932)``
+
+Doc-only
+~~~~~~~~
+
+* ``Update standard provider doc operators in core operators-and-hooks-ref.rst (#49401)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Avoid committing history for providers (#49907)``
+
 1.26.0
 ......
 

apache_airflow_providers_common_sql-1.27.0rc1/docs/commits.rst

@@ -0,0 +1,35 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+ .. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+ .. IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+    `PROVIDER_COMMITS_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+
+ .. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN!
+
+Package apache-airflow-providers-common-sql
+------------------------------------------------------
+
+`Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
+
+
+This is detailed commit list of changes for versions provider package: ``common.sql``.
+For high-level changelog, see :doc:`package information including changelog <index>`.
+
+.. airflow-providers-commits::

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/docs/index.rst

@@ -79,7 +79,7 @@ apache-airflow-providers-common-sql package
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
 
 
-Release: 1.26.0
+Release: 1.27.0
 
 Provider package
 ----------------
@@ -97,12 +97,12 @@ For the minimum Airflow version supported, see ``Requirements`` below.
 Requirements
 ------------
 
-The minimum Apache Airflow version supported by this provider distribution is ``2.9.0``.
+The minimum Apache Airflow version supported by this provider distribution is ``2.10.0``.
 
 ==================  ==================
 PIP package         Version required
 ==================  ==================
-``apache-airflow``  ``>=2.9.0``
+``apache-airflow``  ``>=2.10.0``
 ``sqlparse``        ``>=0.5.1``
 ``more-itertools``  ``>=9.0.0``
 ``methodtools``     ``>=0.4.7``
@@ -133,5 +133,5 @@ Downloading official packages
 You can download officially released packages and verify their checksums and signatures from the
 `Official Apache Download site <https://downloads.apache.org/airflow/providers/>`_
 
-* `The apache-airflow-providers-common-sql 1.26.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.26.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.26.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.26.0.tar.gz.sha512>`__)
-* `The apache-airflow-providers-common-sql 1.26.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.26.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.26.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.26.0-py3-none-any.whl.sha512>`__)
+* `The apache-airflow-providers-common-sql 1.27.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.0.tar.gz.sha512>`__)
+* `The apache-airflow-providers-common-sql 1.27.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.0-py3-none-any.whl.sha512>`__)

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/docs/operators.rst

@@ -173,3 +173,17 @@ The below example demonstrates how to instantiate the SQLThresholdCheckOperator
     :end-before: [END howto_operator_sql_threshold_check]
 
 If the value returned by the query, is within the thresholds, the task passes. Otherwise, it fails.
+
+.. _howto/operator:GenericTransfer:
+
+Generic Transfer
+~~~~~~~~~~~~~~~~
+
+Use the :class:`~airflow.providers.common.sql.operators.generic_transfer.GenericTransfer` to transfer data between
+between two connections.
+
+.. exampleinclude:: /../tests/system/common/sql/example_generic_transfer.py
+    :language: python
+    :dedent: 4
+    :start-after: [START howto_operator_generic_transfer]
+    :end-before: [END howto_operator_generic_transfer]

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/provider.yaml

@@ -22,9 +22,10 @@ description: |
     `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
 
 state: ready
-source-date-epoch: 1744788973
+source-date-epoch: 1746344573
 # note that those versions are maintained by release manager - do not update them manually
 versions:
+  - 1.27.0
   - 1.26.0
   - 1.25.0
   - 1.24.1

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-common-sql"
-version = "1.26.0.rc1"
+version = "1.27.0rc1"
 description = "Provider package apache-airflow-providers-common-sql for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -57,7 +57,7 @@ requires-python = "~=3.9"
 # Make sure to run ``breeze static-checks --type update-providers-dependencies --all-files``
 # After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
 dependencies = [
-    "apache-airflow>=2.9.0rc0",
+    "apache-airflow>=2.10.0rc1",
     "sqlparse>=0.5.1",
     "more-itertools>=9.0.0",
     # The methodtools dependency is necessary since the introduction of dialects:
@@ -122,8 +122,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0"
-"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.26.0/changelog.html"
+"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0"
+"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/src/airflow/providers/common/sql/__init__.py

@@ -29,11 +29,11 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "1.26.0"
+__version__ = "1.27.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
-    "2.9.0"
+    "2.10.0"
 ):
     raise RuntimeError(
-        f"The package `apache-airflow-providers-common-sql:{__version__}` needs Apache Airflow 2.9.0+"
+        f"The package `apache-airflow-providers-common-sql:{__version__}` needs Apache Airflow 2.10.0+"
     )

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/src/airflow/providers/common/sql/hooks/sql.py

@@ -37,8 +37,8 @@ import sqlparse
 from deprecated import deprecated
 from methodtools import lru_cache
 from more_itertools import chunked
-from sqlalchemy import create_engine
-from sqlalchemy.engine import Inspector, make_url
+from sqlalchemy import create_engine, inspect
+from sqlalchemy.engine import make_url
 from sqlalchemy.exc import ArgumentError, NoSuchModuleError
 from typing_extensions import Literal
 
@@ -56,7 +56,7 @@ from airflow.utils.module_loading import import_string
 if TYPE_CHECKING:
     from pandas import DataFrame
     from polars import DataFrame as PolarsDataFrame
-    from sqlalchemy.engine import URL
+    from sqlalchemy.engine import URL, Engine, Inspector
 
     from airflow.models import Connection
     from airflow.providers.openlineage.extractors import OperatorLineage
@@ -307,7 +307,7 @@ class DbApiHook(BaseHook):
             msg = "`sqlalchemy_url` property should be implemented in the provider subclass."
         raise NotImplementedError(msg)
 
-    def get_sqlalchemy_engine(self, engine_kwargs=None):
+    def get_sqlalchemy_engine(self, engine_kwargs=None) -> Engine:
         """
         Get an sqlalchemy_engine object.
 
@@ -328,7 +328,7 @@ class DbApiHook(BaseHook):
 
     @property
     def inspector(self) -> Inspector:
-        return Inspector.from_engine(self.get_sqlalchemy_engine())
+        return inspect(self.get_sqlalchemy_engine())
 
     @cached_property
     def dialect_name(self) -> str:
@@ -890,7 +890,14 @@ class DbApiHook(BaseHook):
                         )
                         sql = self._generate_insert_sql(table, values[0], target_fields, replace, **kwargs)
                         self.log.debug("Generated sql: %s", sql)
-                        cur.executemany(sql, values)
+
+                        try:
+                            cur.executemany(sql, values)
+                        except Exception as e:
+                            self.log.error("Generated sql: %s", sql)
+                            self.log.error("Parameters: %s", values)
+                            raise e
+
                         conn.commit()
                         nb_rows += len(chunked_rows)
                         self.log.info("Loaded %s rows into %s so far", nb_rows, table)
@@ -899,7 +906,14 @@ class DbApiHook(BaseHook):
                         values = self._serialize_cells(row, conn)
                         sql = self._generate_insert_sql(table, values, target_fields, replace, **kwargs)
                         self.log.debug("Generated sql: %s", sql)
-                        cur.execute(sql, values)
+
+                        try:
+                            cur.execute(sql, values)
+                        except Exception as e:
+                            self.log.error("Generated sql: %s", sql)
+                            self.log.error("Parameters: %s", values)
+                            raise e
+
                         if commit_every and i % commit_every == 0:
                             conn.commit()
                             self.log.info("Loaded %s rows into %s so far", i, table)

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/src/airflow/providers/common/sql/hooks/sql.pyi

@@ -37,8 +37,10 @@ from functools import cached_property as cached_property
 from typing import Any, Callable, Protocol, TypeVar, overload
 
 from _typeshed import Incomplete as Incomplete
-from pandas import DataFrame as DataFrame
-from sqlalchemy.engine import URL as URL, Inspector as Inspector
+from pandas import DataFrame as PandasDataFrame
+from polars import DataFrame as PolarsDataFrame
+from sqlalchemy.engine import URL as URL, Engine as Engine, Inspector as Inspector
+from typing_extensions import Literal
 
 from airflow.hooks.base import BaseHook as BaseHook
 from airflow.models import Connection as Connection
@@ -93,7 +95,7 @@ class DbApiHook(BaseHook):
     def get_uri(self) -> str: ...
     @property
     def sqlalchemy_url(self) -> URL: ...
-    def get_sqlalchemy_engine(self, engine_kwargs: Incomplete | None = None): ...
+    def get_sqlalchemy_engine(self, engine_kwargs: Incomplete | None = None) -> Engine: ...
     @property
     def inspector(self) -> Inspector: ...
     @cached_property
@@ -105,10 +107,65 @@ class DbApiHook(BaseHook):
     def get_reserved_words(self, dialect_name: str) -> set[str]: ...
     def get_pandas_df(
         self, sql, parameters: list | tuple | Mapping[str, Any] | None = None, **kwargs
-    ) -> DataFrame: ...
+    ) -> PandasDataFrame: ...
     def get_pandas_df_by_chunks(
         self, sql, parameters: list | tuple | Mapping[str, Any] | None = None, *, chunksize: int, **kwargs
-    ) -> Generator[DataFrame, None, None]: ...
+    ) -> Generator[PandasDataFrame, None, None]: ...
+    @overload
+    def get_df(
+        self,
+        sql: str | list[str],
+        parameters: list | tuple | Mapping[str, Any] | None = None,
+        *,
+        df_type: Literal["pandas"] = "pandas",
+        **kwargs: Any,
+    ) -> PandasDataFrame: ...
+    @overload
+    def get_df(
+        self,
+        sql: str | list[str],
+        parameters: list | tuple | Mapping[str, Any] | None = None,
+        *,
+        df_type: Literal["polars"] = "polars",
+        **kwargs: Any,
+    ) -> PolarsDataFrame: ...
+    @overload
+    def get_df(  # fallback overload
+        self,
+        sql: str | list[str],
+        parameters: list | tuple | Mapping[str, Any] | None = None,
+        *,
+        df_type: Literal["pandas", "polars"] = "pandas",
+    ) -> PandasDataFrame | PolarsDataFrame: ...
+    @overload
+    def get_df_by_chunks(
+        self,
+        sql,
+        parameters: list | tuple | Mapping[str, Any] | None = None,
+        *,
+        chunksize: int,
+        df_type: Literal["pandas"] = "pandas",
+        **kwargs,
+    ) -> Generator[PandasDataFrame, None, None]: ...
+    @overload
+    def get_df_by_chunks(
+        self,
+        sql,
+        parameters: list | tuple | Mapping[str, Any] | None = None,
+        *,
+        chunksize: int,
+        df_type: Literal["polars"],
+        **kwargs,
+    ) -> Generator[PolarsDataFrame, None, None]: ...
+    @overload
+    def get_df_by_chunks(  # fallback overload
+        self,
+        sql,
+        parameters: list | tuple | Mapping[str, Any] | None = None,
+        *,
+        chunksize: int,
+        df_type: Literal["pandas", "polars"] = "pandas",
+    ) -> Generator[PandasDataFrame | PolarsDataFrame, None, None]: ...
     def get_records(
         self, sql: str | list[str], parameters: Iterable | Mapping[str, Any] | None = None
     ) -> Any: ...

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/src/airflow/providers/common/sql/operators/sql.py

@@ -161,7 +161,12 @@ class BaseSQLOperator(BaseOperator):
         :param hook_params: hook parameters
         :return: default hook for this connection
         """
+        hook_params = hook_params or {}
         connection = BaseHook.get_connection(conn_id)
+        conn_params = connection.extra_dejson
+        for conn_param in conn_params:
+            if conn_param not in hook_params:
+                hook_params[conn_param] = conn_params[conn_param]
         return connection.get_hook(hook_params=hook_params)
 
     @cached_property

apache_airflow_providers_common_sql-1.27.0rc1/tests/system/common/sql/example_generic_transfer.py

@@ -0,0 +1,62 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from airflow import DAG
+from airflow.providers.common.sql.operators.generic_transfer import GenericTransfer
+from airflow.utils import timezone
+
+connection_args = {
+    "conn_id": "airflow_db",
+    "conn_type": "Postgres",
+    "host": "postgres",
+    "schema": "postgres",
+    "login": "postgres",
+    "password": "postgres",
+    "port": 5432,
+}
+
+with DAG(
+    "example_generic_transfer",
+    description="Example DAG for GenericTransfer.",
+    default_args=connection_args,
+    start_date=timezone.datetime(2021, 1, 1),
+    schedule=None,
+    catchup=False,
+) as dag:
+    # [START howto_operator_generic_transfer]
+    sql = "SELECT * FROM connection LIMIT 10;"
+
+    generic_transfer = GenericTransfer(
+        task_id="generic_transfer",
+        preoperator=[
+            "DROP TABLE IF EXISTS test_mysql_to_mysql",
+            "CREATE TABLE IF NOT EXISTS test_mysql_to_mysql LIKE connection",
+        ],
+        source_conn_id="airflow_db",
+        destination_conn_id="airflow_db",
+        destination_table="test_mysql_to_mysql",
+        sql=sql,
+    )
+    # [END howto_operator_generic_transfer]
+
+
+from tests_common.test_utils.system_tests import get_test_run  # noqa: E402
+
+# Needed to run the example DAG with pytest (see: tests/system/README.md#run_via_pytest)
+test_run = get_test_run(dag)

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/tests/unit/common/sql/hooks/test_dbapi.py

@@ -267,6 +267,29 @@ class TestDbApiHook:
 
         self.cur.executemany.assert_any_call(sql, rows)
 
+    def test_insert_rows_logs_generated_sql_on_exception(self, caplog):
+        table = "table"
+        rows = [("What's",), ("up",), ("world",)]
+
+        with caplog.at_level(logging.ERROR):
+            self.cur.executemany.side_effect = Exception("Boom!")
+            self.db_hook.supports_executemany = True
+
+            with pytest.raises(Exception, match="Boom!"):
+                self.db_hook.insert_rows(table, iter(rows))
+
+        assert self.conn.close.call_count == 1
+        assert self.cur.close.call_count == 1
+        assert self.conn.commit.call_count == 1
+
+        sql = f"INSERT INTO {table}  VALUES (%s)"
+
+        assert len(caplog.messages) == 2
+        assert any(f"Generated sql: {sql}" in message for message in caplog.messages)
+        assert any(f"Parameters: {rows}" in message for message in caplog.messages)
+
+        self.cur.executemany.assert_any_call(sql, rows)
+
     def test_get_uri_schema_not_none(self):
         self.db_hook.get_connection = mock.MagicMock(
             return_value=Connection(

{apache_airflow_providers_common_sql-1.26.0rc1 → apache_airflow_providers_common_sql-1.27.0rc1}/tests/unit/common/sql/operators/test_sql.py

@@ -113,6 +113,11 @@ class TestBaseSQLOperator:
 
 
 class TestSQLExecuteQueryOperator:
+    def setup_method(self):
+        self.task_id = "test_task"
+        self.conn_id = "sql_default"
+        self._operator = SQLExecuteQueryOperator(task_id=self.task_id, conn_id=self.conn_id, sql="sql")
+
     def _construct_operator(self, sql, **kwargs):
         dag = DAG("test_dag", schedule=None, start_date=datetime.datetime(2017, 1, 1))
         return SQLExecuteQueryOperator(
@@ -190,6 +195,23 @@ class TestSQLExecuteQueryOperator:
         assert descriptions == ("id", "name")
         assert result == [(1, "Alice"), (2, "Bob")]
 
+    @skip_if_force_lowest_dependencies_marker
+    def test_sql_operator_extra_dejson_fields_to_hook_params(self):
+        with mock.patch(
+            "airflow.providers.common.sql.operators.sql.BaseHook.get_connection",
+            return_value=Connection(conn_id="sql_default", conn_type="postgres"),
+        ) as mock_get_conn:
+            mock_get_conn.return_value = Connection(
+                conn_id="google_cloud_bigquery_default",
+                conn_type="gcpbigquery",
+                extra={"use_legacy_sql": False, "priority": "INTERACTIVE"},
+            )
+            self._operator.hook_params = {"use_legacy_sql": True, "location": "us-east1"}
+            assert self._operator._hook.conn_type == "gcpbigquery"
+            assert self._operator._hook.use_legacy_sql is True
+            assert self._operator._hook.location == "us-east1"
+            assert self._operator._hook.priority == "INTERACTIVE"
+
 
 class TestColumnCheckOperator:
     valid_column_mapping = {