apache-airflow-providers-common-sql 1.27.1rc1__tar.gz → 1.27.2rc1__tar.gz

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-common-sql
-Version: 1.27.1rc1
+Version: 1.27.2rc1
 Summary: Provider package apache-airflow-providers-common-sql for Apache Airflow
 Keywords: airflow-provider,common.sql,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -28,8 +28,8 @@ Requires-Dist: apache-airflow-providers-openlineage ; extra == "openlineage"
 Requires-Dist: pandas>=2.1.2,<2.2 ; extra == "pandas"
 Requires-Dist: polars>=1.26.0 ; extra == "polars"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.1/changelog.html
-Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.1
+Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.2/changelog.html
+Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.2
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -63,7 +63,7 @@ Provides-Extra: polars
 
 Package ``apache-airflow-providers-common-sql``
 
-Release: ``1.27.1``
+Release: ``1.27.2``
 
 
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
@@ -76,7 +76,7 @@ This is a provider package for ``common.sql`` provider. All classes for this pro
 are in ``airflow.providers.common.sql`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.2/>`_.
 
 Installation
 ------------
@@ -119,5 +119,5 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.2/changelog.html>`_.
 

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/README.rst

@@ -23,7 +23,7 @@
 
 Package ``apache-airflow-providers-common-sql``
 
-Release: ``1.27.1``
+Release: ``1.27.2``
 
 
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
@@ -36,7 +36,7 @@ This is a provider package for ``common.sql`` provider. All classes for this pro
 are in ``airflow.providers.common.sql`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.2/>`_.
 
 Installation
 ------------
@@ -79,4 +79,4 @@ Dependent package
 ==============================================================================================================  ===============
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.2/changelog.html>`_.

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/docs/changelog.rst

@@ -25,6 +25,22 @@
 Changelog
 ---------
 
+1.27.2
+......
+
+Bug Fixes
+~~~~~~~~~
+
+* ``fix: Use source hook instead of destination hook when reading records in non-paginated mode in GenericTransfer (#50598)``
+
+Misc
+~~~~
+
+* ``Port ''ti.run'' to Task SDK execution path (#50141)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+
 1.27.1
 ......
 

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/docs/index.rst

@@ -79,7 +79,7 @@ apache-airflow-providers-common-sql package
 `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
 
 
-Release: 1.27.1
+Release: 1.27.2
 
 Provider package
 ----------------
@@ -133,5 +133,5 @@ Downloading official packages
 You can download officially released packages and verify their checksums and signatures from the
 `Official Apache Download site <https://downloads.apache.org/airflow/providers/>`_
 
-* `The apache-airflow-providers-common-sql 1.27.1 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.1.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.1.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.1.tar.gz.sha512>`__)
-* `The apache-airflow-providers-common-sql 1.27.1 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.1-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.1-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.1-py3-none-any.whl.sha512>`__)
+* `The apache-airflow-providers-common-sql 1.27.2 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.2.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.2.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.2.tar.gz.sha512>`__)
+* `The apache-airflow-providers-common-sql 1.27.2 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.2-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.2-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_common_sql-1.27.2-py3-none-any.whl.sha512>`__)

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/provider.yaml

@@ -22,12 +22,13 @@ description: |
     `Common SQL Provider <https://en.wikipedia.org/wiki/SQL>`__
 
 state: ready
-source-date-epoch: 1747132237
+source-date-epoch: 1749896343
 # Note that those versions are maintained by release manager - do not update them manually
 # with the exception of case where other provider in sources has >= new provider version.
 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
 # to be done in the same PR
 versions:
+  - 1.27.2
   - 1.27.1
   - 1.27.0
   - 1.26.0

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-common-sql"
-version = "1.27.1rc1"
+version = "1.27.2rc1"
 description = "Provider package apache-airflow-providers-common-sql for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -122,8 +122,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.1"
-"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.1/changelog.html"
+"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.2"
+"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-common-sql/1.27.2/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/src/airflow/providers/common/sql/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "1.27.1"
+__version__ = "1.27.2"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.10.0"

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/src/airflow/providers/common/sql/operators/generic_transfer.py

@@ -162,7 +162,7 @@ class GenericTransfer(BaseOperator):
             self.log.info("Extracting data from %s", self.source_conn_id)
             self.log.info("Executing: \n %s", self.sql)
 
-            results = self.destination_hook.get_records(self.sql)
+            results = self.source_hook.get_records(self.sql)
 
             self.log.info("Inserting rows into %s", self.destination_conn_id)
             self.destination_hook.insert_rows(table=self.destination_table, rows=results, **self.insert_args)

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/tests/unit/common/sql/operators/test_generic_transfer.py

@@ -24,6 +24,7 @@ from unittest import mock
 from unittest.mock import MagicMock
 
 import pytest
+from more_itertools import flatten
 
 from airflow.exceptions import AirflowProviderDeprecationWarning
 from airflow.models.connection import Connection
@@ -34,7 +35,7 @@ from airflow.providers.postgres.hooks.postgres import PostgresHook
 from airflow.utils import timezone
 
 from tests_common.test_utils.compat import GenericTransfer
-from tests_common.test_utils.operators.run_deferrable import execute_operator
+from tests_common.test_utils.operators.run_deferrable import execute_operator, mock_context
 from tests_common.test_utils.providers import get_provider_min_airflow_version
 
 pytestmark = pytest.mark.db_test
@@ -43,6 +44,12 @@ DEFAULT_DATE = timezone.datetime(2015, 1, 1)
 DEFAULT_DATE_ISO = DEFAULT_DATE.isoformat()
 DEFAULT_DATE_DS = DEFAULT_DATE_ISO[:10]
 TEST_DAG_ID = "unit_test_dag"
+INSERT_ARGS = {
+    "commit_every": 1000,  # Number of rows inserted in each batch
+    "executemany": True,  # Enable batch inserts
+    "fast_executemany": True,  # Boost performance for MSSQL inserts
+    "replace": True,  # Used for upserts/merges if needed
+}
 counter = 0
 
 
@@ -175,6 +182,44 @@ class TestPostgres:
 
 
 class TestGenericTransfer:
+    mocked_source_hook = MagicMock(conn_name_attr="my_source_conn_id", spec=DbApiHook)
+    mocked_destination_hook = MagicMock(conn_name_attr="my_destination_conn_id", spec=DbApiHook)
+    mocked_hooks = {
+        "my_source_conn_id": mocked_source_hook,
+        "my_destination_conn_id": mocked_destination_hook,
+    }
+
+    @classmethod
+    def get_hook(cls, conn_id: str, hook_params: dict | None = None):
+        return cls.mocked_hooks[conn_id]
+
+    @classmethod
+    def get_connection(cls, conn_id: str):
+        mocked_hook = cls.get_hook(conn_id=conn_id)
+        mocked_conn = MagicMock(conn_id=conn_id, spec=Connection)
+        mocked_conn.get_hook.return_value = mocked_hook
+        return mocked_conn
+
+    def setup_method(self):
+        # Reset mock states before each test
+        self.mocked_source_hook.reset_mock()
+        self.mocked_destination_hook.reset_mock()
+
+        # Set up the side effect for paginated read
+        records = [
+            [[1, 2], [11, 12], [3, 4], [13, 14]],
+            [[3, 4], [13, 14]],
+        ]
+
+        def get_records_side_effect(sql: str):
+            if records:
+                if "LIMIT" not in sql:
+                    return list(flatten(records))
+                return records.pop(0)
+            return []
+
+        self.mocked_source_hook.get_records.side_effect = get_records_side_effect
+
     def test_templated_fields(self):
         dag = DAG(
             "test_dag",
@@ -209,40 +254,37 @@ class TestGenericTransfer:
         assert operator.preoperator == "my_preoperator"
         assert operator.insert_args == {"commit_every": 5000, "executemany": True, "replace": True}
 
+    def test_non_paginated_read(self):
+        with mock.patch("airflow.hooks.base.BaseHook.get_connection", side_effect=self.get_connection):
+            with mock.patch("airflow.hooks.base.BaseHook.get_hook", side_effect=self.get_hook):
+                operator = GenericTransfer(
+                    task_id="transfer_table",
+                    source_conn_id="my_source_conn_id",
+                    destination_conn_id="my_destination_conn_id",
+                    sql="SELECT * FROM HR.EMPLOYEES",
+                    destination_table="NEW_HR.EMPLOYEES",
+                    insert_args=INSERT_ARGS,
+                    execution_timeout=timedelta(hours=1),
+                )
+
+                operator.execute(context=mock_context(task=operator))
+
+        assert self.mocked_source_hook.get_records.call_count == 1
+        assert self.mocked_source_hook.get_records.call_args_list[0].args[0] == "SELECT * FROM HR.EMPLOYEES"
+        assert self.mocked_destination_hook.insert_rows.call_count == 1
+        assert self.mocked_destination_hook.insert_rows.call_args_list[0].kwargs == {
+            **INSERT_ARGS,
+            **{"rows": [[1, 2], [11, 12], [3, 4], [13, 14], [3, 4], [13, 14]], "table": "NEW_HR.EMPLOYEES"},
+        }
+
     def test_paginated_read(self):
         """
         This unit test is based on the example described in the medium article:
         https://medium.com/apache-airflow/transfering-data-from-sap-hana-to-mssql-using-the-airflow-generictransfer-d29f147a9f1f
         """
 
-        def create_get_records_side_effect():
-            records = [
-                [[1, 2], [11, 12], [3, 4], [13, 14]],
-                [[3, 4], [13, 14]],
-            ]
-
-            def side_effect(sql: str):
-                if records:
-                    return records.pop(0)
-                return []
-
-            return side_effect
-
-        get_records_side_effect = create_get_records_side_effect()
-
-        def get_hook(conn_id: str, hook_params: dict | None = None):
-            mocked_hook = MagicMock(conn_name_attr=conn_id, spec=DbApiHook)
-            mocked_hook.get_records.side_effect = get_records_side_effect
-            return mocked_hook
-
-        def get_connection(conn_id: str):
-            mocked_hook = get_hook(conn_id=conn_id)
-            mocked_conn = MagicMock(conn_id=conn_id, spec=Connection)
-            mocked_conn.get_hook.return_value = mocked_hook
-            return mocked_conn
-
-        with mock.patch("airflow.hooks.base.BaseHook.get_connection", side_effect=get_connection):
-            with mock.patch("airflow.hooks.base.BaseHook.get_hook", side_effect=get_hook):
+        with mock.patch("airflow.hooks.base.BaseHook.get_connection", side_effect=self.get_connection):
+            with mock.patch("airflow.hooks.base.BaseHook.get_hook", side_effect=self.get_hook):
                 operator = GenericTransfer(
                     task_id="transfer_table",
                     source_conn_id="my_source_conn_id",
@@ -250,12 +292,7 @@ class TestGenericTransfer:
                     sql="SELECT * FROM HR.EMPLOYEES",
                     destination_table="NEW_HR.EMPLOYEES",
                     page_size=1000,  # Fetch data in chunks of 1000 rows for pagination
-                    insert_args={
-                        "commit_every": 1000,  # Number of rows inserted in each batch
-                        "executemany": True,  # Enable batch inserts
-                        "fast_executemany": True,  # Boost performance for MSSQL inserts
-                        "replace": True,  # Used for upserts/merges if needed
-                    },
+                    insert_args=INSERT_ARGS,
                     execution_timeout=timedelta(hours=1),
                 )
 
@@ -267,6 +304,21 @@ class TestGenericTransfer:
                 assert events[1].payload["results"] == [[3, 4], [13, 14]]
                 assert not events[2].payload["results"]
 
+        assert self.mocked_source_hook.get_records.call_count == 3
+        assert (
+            self.mocked_source_hook.get_records.call_args_list[0].args[0]
+            == "SELECT * FROM HR.EMPLOYEES LIMIT 1000 OFFSET 0"
+        )
+        assert self.mocked_destination_hook.insert_rows.call_count == 2
+        assert self.mocked_destination_hook.insert_rows.call_args_list[0].kwargs == {
+            **INSERT_ARGS,
+            **{"rows": [[1, 2], [11, 12], [3, 4], [13, 14]], "table": "NEW_HR.EMPLOYEES"},
+        }
+        assert self.mocked_destination_hook.insert_rows.call_args_list[1].kwargs == {
+            **INSERT_ARGS,
+            **{"rows": [[3, 4], [13, 14]], "table": "NEW_HR.EMPLOYEES"},
+        }
+
     def test_when_provider_min_airflow_version_is_3_0_or_higher_remove_obsolete_method(self):
         """
         Once this test starts failing due to the fact that the minimum Airflow version is now 3.0.0 or higher

{apache_airflow_providers_common_sql-1.27.1rc1 → apache_airflow_providers_common_sql-1.27.2rc1}/tests/unit/common/sql/operators/test_sql.py

@@ -48,7 +48,7 @@ from airflow.utils.types import DagRunType
 
 from tests_common.test_utils.markers import skip_if_force_lowest_dependencies_marker
 from tests_common.test_utils.providers import get_provider_min_airflow_version
-from tests_common.test_utils.version_compat import AIRFLOW_V_3_0_PLUS
+from tests_common.test_utils.version_compat import AIRFLOW_V_3_0_1, AIRFLOW_V_3_0_PLUS
 
 if AIRFLOW_V_3_0_PLUS:
     from airflow.models.xcom import XComModel as XCom
@@ -1224,7 +1224,7 @@ class TestSqlBranch:
 
         mock_get_records.return_value = 1
 
-        if AIRFLOW_V_3_0_PLUS:
+        if AIRFLOW_V_3_0_1:
             from airflow.exceptions import DownstreamTasksSkipped
 
             with pytest.raises(DownstreamTasksSkipped) as exc_info:
@@ -1272,7 +1272,7 @@ class TestSqlBranch:
         mock_get_records = mock_get_db_hook.return_value.get_first
         mock_get_records.return_value = true_value
 
-        if AIRFLOW_V_3_0_PLUS:
+        if AIRFLOW_V_3_0_1:
             from airflow.exceptions import DownstreamTasksSkipped
 
             with pytest.raises(DownstreamTasksSkipped) as exc_info:
@@ -1320,7 +1320,7 @@ class TestSqlBranch:
         mock_get_records = mock_get_db_hook.return_value.get_first
 
         mock_get_records.return_value = false_value
-        if AIRFLOW_V_3_0_PLUS:
+        if AIRFLOW_V_3_0_1:
             from airflow.exceptions import DownstreamTasksSkipped
 
             with pytest.raises(DownstreamTasksSkipped) as exc_info:
@@ -1377,7 +1377,7 @@ class TestSqlBranch:
         mock_get_records = mock_get_db_hook.return_value.get_first
         mock_get_records.return_value = [["1"]]
 
-        if AIRFLOW_V_3_0_PLUS:
+        if AIRFLOW_V_3_0_1:
             from airflow.exceptions import DownstreamTasksSkipped
 
             with pytest.raises(DownstreamTasksSkipped) as exc_info:
@@ -1495,7 +1495,7 @@ class TestSqlBranch:
         mock_get_records = mock_get_db_hook.return_value.get_first
         mock_get_records.return_value = [false_value]
 
-        if AIRFLOW_V_3_0_PLUS:
+        if AIRFLOW_V_3_0_1:
             from airflow.exceptions import DownstreamTasksSkipped
 
             with pytest.raises(DownstreamTasksSkipped) as exc_info: