apache-airflow-providers-amazon 9.9.1rc1__tar.gz → 9.10.0rc1__tar.gz

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/PKG-INFO

@@ -1,11 +1,11 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-amazon
-Version: 9.9.1rc1
+Version: 9.10.0rc1
 Summary: Provider package apache-airflow-providers-amazon for Apache Airflow
 Keywords: airflow-provider,amazon,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
 Maintainer-email: Apache Software Foundation <dev@airflow.apache.org>
-Requires-Python: ~=3.10
+Requires-Python: >=3.10
 Description-Content-Type: text/x-rst
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
@@ -54,8 +54,8 @@ Requires-Dist: apache-airflow-providers-salesforce ; extra == "salesforce"
 Requires-Dist: apache-airflow-providers-ssh ; extra == "ssh"
 Requires-Dist: apache-airflow-providers-standard ; extra == "standard"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.1/changelog.html
-Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.1
+Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html
+Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.10.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -103,7 +103,7 @@ Provides-Extra: standard
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``9.9.1``
+Release: ``9.10.0``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -116,7 +116,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/>`_.
 
 Installation
 ------------
@@ -184,5 +184,5 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html>`_.
 

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/README.rst

@@ -23,7 +23,7 @@
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``9.9.1``
+Release: ``9.10.0``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -36,7 +36,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/>`_.
 
 Installation
 ------------
@@ -104,4 +104,4 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html>`_.

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/docs/changelog.rst

@@ -26,14 +26,21 @@
 Changelog
 ---------
 
-9.9.1
-.....
+9.10.0
+......
+
+Features
+~~~~~~~~
+
+* ``Add support for S3 dag bundle (#46621)``
 
 Bug Fixes
 ~~~~~~~~~
 
 * ``Fix GlueJobOperator deferred waiting (#52314)``
 * ``Handle exceptions when fetching status in GlueJobHook (#52262)``
+* ``Handle 'S3KeySensor' in 'deferrable' mode splits 'bucket_key' into individual chars (#52983)``
+* ``Pass the region_name from the GlueJobOperator / GlueJobSensor to the Trigger (#52904)``
 
 Misc
 ~~~~
@@ -50,6 +57,14 @@ Misc
 * ``Use base AWS classes in Glue Trigger / Sensor and implement custom waiter (#52243)``
 * ``Add Airflow 3.0+ Task SDK support to AWS Batch Executor (#52121)``
 * ``Refactor operator_extra_links property in BatchOperator (#51385)``
+* ``Remove unused batch methods from auth manager (#52883)``
+* ``Add debug logging for endpoint_url in AWS Connection (#52856)``
+* ``Remove 'MENU' from 'ResourceMethod' in auth manager (#52731)``
+* ``Remove 'MAX_XCOM_SIZE' hardcoded constant from Airflow core  (#52978)``
+* ``More robust handling of 'BaseHook.get_connection''s 'CONNECTION_NOT_FOUND' Task SDK exception (#52838)``
+* ``Move all BaseHook usages to version_compat in Amazon (#52796)``
+* ``Remove upper-binding for "python-requires" (#52980)``
+* ``Temporarily switch to use >=,< pattern instead of '~=' (#52967)``
 
 Doc-only
 ~~~~~~~~
@@ -66,6 +81,7 @@ Doc-only
    * ``Introducing fixture to create 'Connections' without DB in provider tests (#51930)``
    * ``Switch the Supervisor/task process from line-based to length-prefixed (#51699)``
    * ``Mocked time.sleep to avoid actual sleep time (#51752)``
+   * ``Prepare release for July 2025 1st provider wave (#52727)``
 
 9.9.0
 .....

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/docs/index.rst

@@ -87,7 +87,7 @@ apache-airflow-providers-amazon package
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
 
 
-Release: 9.9.1
+Release: 9.10.0
 
 Provider package
 ----------------
@@ -166,5 +166,5 @@ Downloading official packages
 You can download officially released packages and verify their checksums and signatures from the
 `Official Apache Download site <https://downloads.apache.org/airflow/providers/>`_
 
-* `The apache-airflow-providers-amazon 9.9.1 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.1.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.1.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.1.tar.gz.sha512>`__)
-* `The apache-airflow-providers-amazon 9.9.1 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.1-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.1-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.1-py3-none-any.whl.sha512>`__)
+* `The apache-airflow-providers-amazon 9.10.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0.tar.gz.sha512>`__)
+* `The apache-airflow-providers-amazon 9.10.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0-py3-none-any.whl.sha512>`__)

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/provider.yaml

@@ -28,7 +28,7 @@ source-date-epoch: 1751472351
 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
 # to be done in the same PR
 versions:
-  - 9.9.1
+  - 9.10.0
   - 9.9.0
   - 9.8.0
   - 9.7.0

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-amazon"
-version = "9.9.1rc1"
+version = "9.10.0rc1"
 description = "Provider package apache-airflow-providers-amazon for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -49,7 +49,7 @@ classifiers = [
     "Programming Language :: Python :: 3.12",
     "Topic :: System :: Monitoring",
 ]
-requires-python = "~=3.10"
+requires-python = ">=3.10"
 
 # The dependencies should be modified in place in the generated file.
 # Any change in the dependencies is preserved when the file is regenerated
@@ -204,8 +204,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.1"
-"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.1/changelog.html"
+"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.10.0"
+"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "9.9.1"
+__version__ = "9.10.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.10.0"

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/auth_manager/avp/facade.py

@@ -37,6 +37,13 @@ from airflow.utils.log.logging_mixin import LoggingMixin
 
 if TYPE_CHECKING:
     from airflow.api_fastapi.auth.managers.base_auth_manager import ResourceMethod
+
+    try:
+        from airflow.api_fastapi.auth.managers.base_auth_manager import ExtendedResourceMethod
+    except ImportError:
+        from airflow.api_fastapi.auth.managers.base_auth_manager import (
+            ResourceMethod as ExtendedResourceMethod,
+        )
     from airflow.providers.amazon.aws.auth_manager.user import AwsAuthManagerUser
 
 
@@ -48,7 +55,7 @@ NB_REQUESTS_PER_BATCH = 30
 class IsAuthorizedRequest(TypedDict, total=False):
     """Represent the parameters of ``is_authorized`` method in AVP facade."""
 
-    method: ResourceMethod
+    method: ExtendedResourceMethod
     entity_type: AvpEntities
     entity_id: str | None
     context: dict | None

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py

@@ -44,10 +44,7 @@ from airflow.providers.amazon.version_compat import AIRFLOW_V_3_0_PLUS
 if TYPE_CHECKING:
     from airflow.api_fastapi.auth.managers.base_auth_manager import ResourceMethod
     from airflow.api_fastapi.auth.managers.models.batch_apis import (
-        IsAuthorizedConnectionRequest,
         IsAuthorizedDagRequest,
-        IsAuthorizedPoolRequest,
-        IsAuthorizedVariableRequest,
     )
     from airflow.api_fastapi.auth.managers.models.resource_details import (
         AccessView,
@@ -247,24 +244,6 @@ class AwsAuthManager(BaseAuthManager[AwsAuthManagerUser]):
 
         return [menu_item for menu_item in menu_items if _has_access_to_menu_item(requests[menu_item.value])]
 
-    def batch_is_authorized_connection(
-        self,
-        requests: Sequence[IsAuthorizedConnectionRequest],
-        *,
-        user: AwsAuthManagerUser,
-    ) -> bool:
-        facade_requests: Sequence[IsAuthorizedRequest] = [
-            {
-                "method": request["method"],
-                "entity_type": AvpEntities.CONNECTION,
-                "entity_id": cast("ConnectionDetails", request["details"]).conn_id
-                if request.get("details")
-                else None,
-            }
-            for request in requests
-        ]
-        return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
-
     def batch_is_authorized_dag(
         self,
         requests: Sequence[IsAuthorizedDagRequest],
@@ -288,40 +267,6 @@ class AwsAuthManager(BaseAuthManager[AwsAuthManagerUser]):
         ]
         return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
 
-    def batch_is_authorized_pool(
-        self,
-        requests: Sequence[IsAuthorizedPoolRequest],
-        *,
-        user: AwsAuthManagerUser,
-    ) -> bool:
-        facade_requests: Sequence[IsAuthorizedRequest] = [
-            {
-                "method": request["method"],
-                "entity_type": AvpEntities.POOL,
-                "entity_id": cast("PoolDetails", request["details"]).name if request.get("details") else None,
-            }
-            for request in requests
-        ]
-        return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
-
-    def batch_is_authorized_variable(
-        self,
-        requests: Sequence[IsAuthorizedVariableRequest],
-        *,
-        user: AwsAuthManagerUser,
-    ) -> bool:
-        facade_requests: Sequence[IsAuthorizedRequest] = [
-            {
-                "method": request["method"],
-                "entity_type": AvpEntities.VARIABLE,
-                "entity_id": cast("VariableDetails", request["details"]).key
-                if request.get("details")
-                else None,
-            }
-            for request in requests
-        ]
-        return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
-
     def filter_authorized_dag_ids(
         self,
         *,

apache_airflow_providers_amazon-9.10.0rc1/src/airflow/providers/amazon/aws/bundles/s3.py

@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import structlog
+
+from airflow.dag_processing.bundles.base import BaseDagBundle
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+
+
+class S3DagBundle(BaseDagBundle):
+    """
+    S3 DAG bundle - exposes a directory in S3 as a DAG bundle.
+
+    This allows Airflow to load DAGs directly from an S3 bucket.
+
+    :param aws_conn_id: Airflow connection ID for AWS.  Defaults to AwsBaseHook.default_conn_name.
+    :param bucket_name: The name of the S3 bucket containing the DAG files.
+    :param prefix:  Optional subdirectory within the S3 bucket where the DAGs are stored.
+                    If None, DAGs are assumed to be at the root of the bucket (Optional).
+    """
+
+    supports_versioning = False
+
+    def __init__(
+        self,
+        *,
+        aws_conn_id: str = AwsBaseHook.default_conn_name,
+        bucket_name: str,
+        prefix: str = "",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.aws_conn_id = aws_conn_id
+        self.bucket_name = bucket_name
+        self.prefix = prefix
+        # Local path where S3 DAGs are downloaded
+        self.s3_dags_dir: Path = self.base_dir
+
+        log = structlog.get_logger(__name__)
+        self._log = log.bind(
+            bundle_name=self.name,
+            version=self.version,
+            bucket_name=self.bucket_name,
+            prefix=self.prefix,
+            aws_conn_id=self.aws_conn_id,
+        )
+        self._s3_hook: S3Hook | None = None
+
+    def _initialize(self):
+        with self.lock():
+            if not self.s3_dags_dir.exists():
+                self._log.info("Creating local DAGs directory: %s", self.s3_dags_dir)
+                os.makedirs(self.s3_dags_dir)
+
+            if not self.s3_dags_dir.is_dir():
+                raise AirflowException(f"Local DAGs path: {self.s3_dags_dir} is not a directory.")
+
+            if not self.s3_hook.check_for_bucket(bucket_name=self.bucket_name):
+                raise AirflowException(f"S3 bucket '{self.bucket_name}' does not exist.")
+
+            if self.prefix:
+                # don't check when prefix is ""
+                if not self.s3_hook.check_for_prefix(
+                    bucket_name=self.bucket_name, prefix=self.prefix, delimiter="/"
+                ):
+                    raise AirflowException(
+                        f"S3 prefix 's3://{self.bucket_name}/{self.prefix}' does not exist."
+                    )
+            self.refresh()
+
+    def initialize(self) -> None:
+        self._initialize()
+        super().initialize()
+
+    @property
+    def s3_hook(self):
+        if self._s3_hook is None:
+            try:
+                self._s3_hook: S3Hook = S3Hook(aws_conn_id=self.aws_conn_id)  # Initialize S3 hook.
+            except AirflowException as e:
+                self._log.warning("Could not create S3Hook for connection %s: %s", self.aws_conn_id, e)
+        return self._s3_hook
+
+    def __repr__(self):
+        return (
+            f"<S3DagBundle("
+            f"name={self.name!r}, "
+            f"bucket_name={self.bucket_name!r}, "
+            f"prefix={self.prefix!r}, "
+            f"version={self.version!r}"
+            f")>"
+        )
+
+    def get_current_version(self) -> str | None:
+        """Return the current version of the DAG bundle. Currently not supported."""
+        return None
+
+    @property
+    def path(self) -> Path:
+        """Return the local path to the DAG files."""
+        return self.s3_dags_dir  # Path where DAGs are downloaded.
+
+    def refresh(self) -> None:
+        """Refresh the DAG bundle by re-downloading the DAGs from S3."""
+        if self.version:
+            raise AirflowException("Refreshing a specific version is not supported")
+
+        with self.lock():
+            self._log.debug(
+                "Downloading DAGs from s3://%s/%s to %s", self.bucket_name, self.prefix, self.s3_dags_dir
+            )
+            self.s3_hook.sync_to_local_dir(
+                bucket_name=self.bucket_name,
+                s3_prefix=self.prefix,
+                local_dir=self.s3_dags_dir,
+                delete_stale=True,
+            )
+
+    def view_url(self, version: str | None = None) -> str | None:
+        """Return a URL for viewing the DAGs in S3. Currently, versioning is not supported."""
+        if self.version:
+            raise AirflowException("S3 url with version is not supported")
+
+        # https://<bucket-name>.s3.<region>.amazonaws.com/<object-key>
+        url = f"https://{self.bucket_name}.s3"
+        if self.s3_hook.region_name:
+            url += f".{self.s3_hook.region_name}"
+        url += ".amazonaws.com"
+        if self.prefix:
+            url += f"/{self.prefix}"
+
+        return url

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/hooks/base_aws.py

@@ -60,13 +60,9 @@ from airflow.exceptions import (
 from airflow.providers.amazon.aws.utils.connection_wrapper import AwsConnectionWrapper
 from airflow.providers.amazon.aws.utils.identifiers import generate_uuid
 from airflow.providers.amazon.aws.utils.suppress import return_on_error
+from airflow.providers.amazon.version_compat import BaseHook
 from airflow.providers.common.compat.version_compat import AIRFLOW_V_3_0_PLUS
 from airflow.providers_manager import ProvidersManager
-
-try:
-    from airflow.sdk import BaseHook
-except ImportError:
-    from airflow.hooks.base import BaseHook  # type: ignore[attr-defined,no-redef]
 from airflow.utils.helpers import exactly_one
 from airflow.utils.log.logging_mixin import LoggingMixin
 
@@ -82,7 +78,7 @@ BaseAwsConnection = TypeVar("BaseAwsConnection", bound=Union[BaseClient, Service
 
 
 if AIRFLOW_V_3_0_PLUS:
-    from airflow.sdk.exceptions import AirflowRuntimeError
+    from airflow.sdk.exceptions import AirflowRuntimeError, ErrorType
 
 if TYPE_CHECKING:
     from aiobotocore.session import AioSession
@@ -623,19 +619,16 @@ class AwsGenericHook(BaseHook, Generic[BaseAwsConnection]):
         """Get the Airflow Connection object and wrap it in helper (cached)."""
         connection = None
         if self.aws_conn_id:
-            possible_exceptions: tuple[type[Exception], ...]
-
-            if AIRFLOW_V_3_0_PLUS:
-                possible_exceptions = (AirflowNotFoundException, AirflowRuntimeError)
-            else:
-                possible_exceptions = (AirflowNotFoundException,)
-
             try:
                 connection = self.get_connection(self.aws_conn_id)
-            except possible_exceptions as e:
-                if isinstance(
-                    e, AirflowNotFoundException
-                ) or f"Connection with ID {self.aws_conn_id} not found" in str(e):
+            except Exception as e:
+                not_found_exc_via_core = isinstance(e, AirflowNotFoundException)
+                not_found_exc_via_task_sdk = (
+                    AIRFLOW_V_3_0_PLUS
+                    and isinstance(e, AirflowRuntimeError)
+                    and e.error.error == ErrorType.CONNECTION_NOT_FOUND
+                )
+                if not_found_exc_via_core or not_found_exc_via_task_sdk:
                     self.log.warning(
                         "Unable to find AWS Connection ID '%s', switching to empty.", self.aws_conn_id
                     )

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/hooks/s3.py

@@ -635,6 +635,10 @@ class S3Hook(AwsBaseHook):
         delimiter: str | None = "/",
     ) -> list[Any]:
         """Get a list of files in the bucket."""
+        # Validate that bucket_keys is in fact a list, otherwise, the characters will be split
+        if isinstance(bucket_keys, str):
+            bucket_keys = [bucket_keys]
+
         keys: list[Any] = []
         for key in bucket_keys:
             prefix = key
@@ -652,7 +656,9 @@ class S3Hook(AwsBaseHook):
             response = paginator.paginate(**params)
             async for page in response:
                 if "Contents" in page:
-                    keys.extend(k for k in page["Contents"] if isinstance(k.get("Size"), (int, float)))
+                    keys.extend(
+                        k.get("Key") for k in page["Contents"] if isinstance(k.get("Size"), (int, float))
+                    )
         return keys
 
     async def _list_keys_async(
@@ -1683,3 +1689,80 @@ class S3Hook(AwsBaseHook):
         """
         s3_client = self.get_conn()
         s3_client.delete_bucket_tagging(Bucket=bucket_name)
+
+    def _sync_to_local_dir_delete_stale_local_files(self, current_s3_objects: list[Path], local_dir: Path):
+        current_s3_keys = {key for key in current_s3_objects}
+
+        for item in local_dir.iterdir():
+            item: Path  # type: ignore[no-redef]
+            absolute_item_path = item.resolve()
+
+            if absolute_item_path not in current_s3_keys:
+                try:
+                    if item.is_file():
+                        item.unlink(missing_ok=True)
+                        self.log.debug("Deleted stale local file: %s", item)
+                    elif item.is_dir():
+                        # delete only when the folder is empty
+                        if not os.listdir(item):
+                            item.rmdir()
+                            self.log.debug("Deleted stale empty directory: %s", item)
+                    else:
+                        self.log.debug("Skipping stale item of unknown type: %s", item)
+                except OSError as e:
+                    self.log.error("Error deleting stale item %s: %s", item, e)
+                    raise e
+
+    def _sync_to_local_dir_if_changed(self, s3_bucket, s3_object, local_target_path: Path):
+        should_download = False
+        download_msg = ""
+        if not local_target_path.exists():
+            should_download = True
+            download_msg = f"Local file {local_target_path} does not exist."
+        else:
+            local_stats = local_target_path.stat()
+
+            if s3_object.size != local_stats.st_size:
+                should_download = True
+                download_msg = (
+                    f"S3 object size ({s3_object.size}) and local file size ({local_stats.st_size}) differ."
+                )
+
+            s3_last_modified = s3_object.last_modified
+            if local_stats.st_mtime < s3_last_modified.microsecond:
+                should_download = True
+                download_msg = f"S3 object last modified ({s3_last_modified.microsecond}) and local file last modified ({local_stats.st_mtime}) differ."
+
+        if should_download:
+            s3_bucket.download_file(s3_object.key, local_target_path)
+            self.log.debug(
+                "%s Downloaded %s to %s", download_msg, s3_object.key, local_target_path.as_posix()
+            )
+        else:
+            self.log.debug(
+                "Local file %s is up-to-date with S3 object %s. Skipping download.",
+                local_target_path.as_posix(),
+                s3_object.key,
+            )
+
+    def sync_to_local_dir(self, bucket_name: str, local_dir: Path, s3_prefix="", delete_stale: bool = True):
+        """Download S3 files from the S3 bucket to the local directory."""
+        self.log.debug("Downloading data from s3://%s/%s to %s", bucket_name, s3_prefix, local_dir)
+
+        local_s3_objects = []
+        s3_bucket = self.get_bucket(bucket_name)
+        for obj in s3_bucket.objects.filter(Prefix=s3_prefix):
+            obj_path = Path(obj.key)
+            local_target_path = local_dir.joinpath(obj_path.relative_to(s3_prefix))
+            if not local_target_path.parent.exists():
+                local_target_path.parent.mkdir(parents=True, exist_ok=True)
+                self.log.debug("Created local directory: %s", local_target_path.parent)
+            self._sync_to_local_dir_if_changed(
+                s3_bucket=s3_bucket, s3_object=obj, local_target_path=local_target_path
+            )
+            local_s3_objects.append(local_target_path)
+
+        if delete_stale:
+            self._sync_to_local_dir_delete_stale_local_files(
+                current_s3_objects=local_s3_objects, local_dir=local_dir
+            )

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/hooks/sagemaker_unified_studio.py

@@ -26,11 +26,7 @@ from sagemaker_studio.sagemaker_studio_api import SageMakerStudioAPI
 
 from airflow.exceptions import AirflowException
 from airflow.providers.amazon.aws.utils.sagemaker_unified_studio import is_local_runner
-
-try:
-    from airflow.sdk import BaseHook
-except ImportError:
-    from airflow.hooks.base import BaseHook  # type: ignore[attr-defined,no-redef]
+from airflow.providers.amazon.version_compat import BaseHook
 
 
 class SageMakerNotebookHook(BaseHook):

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/operators/glue.py

@@ -249,6 +249,7 @@ class GlueJobOperator(AwsBaseOperator[GlueJobHook]):
                     aws_conn_id=self.aws_conn_id,
                     waiter_delay=self.waiter_delay,
                     waiter_max_attempts=self.waiter_max_attempts,
+                    region_name=self.region_name,
                 ),
                 method_name="execute_complete",
             )

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/sensors/glue.py

@@ -105,6 +105,7 @@ class GlueJobSensor(AwsBaseSensor[GlueJobHook]):
                     aws_conn_id=self.aws_conn_id,
                     waiter_delay=int(self.poke_interval),
                     waiter_max_attempts=self.max_retries,
+                    region_name=self.region_name,
                 ),
                 method_name="execute_complete",
             )

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/transfers/google_api_to_s3.py

@@ -24,7 +24,7 @@ import sys
 from collections.abc import Sequence
 from typing import TYPE_CHECKING
 
-from airflow.models.xcom import MAX_XCOM_SIZE, XCOM_RETURN_KEY
+from airflow.models.xcom import XCOM_RETURN_KEY
 from airflow.providers.amazon.aws.hooks.s3 import S3Hook
 from airflow.providers.amazon.version_compat import BaseOperator
 from airflow.providers.google.common.hooks.discovery_api import GoogleDiscoveryApiHook
@@ -36,6 +36,10 @@ if TYPE_CHECKING:
         from airflow.models import TaskInstance as RuntimeTaskInstanceProtocol  # type: ignore[assignment]
     from airflow.utils.context import Context
 
+# MAX XCOM Size is 48KB
+# https://github.com/apache/airflow/pull/1618#discussion_r68249677
+MAX_XCOM_SIZE = 49344
+
 
 class GoogleApiToS3Operator(BaseOperator):
     """

{apache_airflow_providers_amazon-9.9.1rc1 → apache_airflow_providers_amazon-9.10.0rc1}/src/airflow/providers/amazon/aws/transfers/s3_to_sql.py

@@ -23,12 +23,7 @@ from typing import TYPE_CHECKING
 
 from airflow.exceptions import AirflowException
 from airflow.providers.amazon.aws.hooks.s3 import S3Hook
-from airflow.providers.amazon.version_compat import BaseOperator
-
-try:
-    from airflow.sdk import BaseHook
-except ImportError:
-    from airflow.hooks.base import BaseHook  # type: ignore[attr-defined,no-redef]
+from airflow.providers.amazon.version_compat import BaseHook, BaseOperator
 
 if TYPE_CHECKING:
     from airflow.utils.context import Context