apache-airflow-providers-amazon 9.9.0rc1__tar.gz → 9.10.0__tar.gz

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/PKG-INFO

@@ -1,11 +1,11 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-amazon
-Version: 9.9.0rc1
+Version: 9.10.0
 Summary: Provider package apache-airflow-providers-amazon for Apache Airflow
 Keywords: airflow-provider,amazon,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
 Maintainer-email: Apache Software Foundation <dev@airflow.apache.org>
-Requires-Python: ~=3.9
+Requires-Python: >=3.10
 Description-Content-Type: text/x-rst
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
@@ -15,14 +15,13 @@ Classifier: Intended Audience :: System Administrators
 Classifier: Framework :: Apache Airflow
 Classifier: Framework :: Apache Airflow :: Provider
 Classifier: License :: OSI Approved :: Apache Software License
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
-Requires-Dist: apache-airflow>=2.10.0rc1
-Requires-Dist: apache-airflow-providers-common-compat>=1.6.1rc1
-Requires-Dist: apache-airflow-providers-common-sql>=1.27.0rc1
+Requires-Dist: apache-airflow>=2.10.0
+Requires-Dist: apache-airflow-providers-common-compat>=1.6.1
+Requires-Dist: apache-airflow-providers-common-sql>=1.27.0
 Requires-Dist: apache-airflow-providers-http
 Requires-Dist: boto3>=1.37.2
 Requires-Dist: botocore>=1.37.2
@@ -37,8 +36,8 @@ Requires-Dist: sagemaker-studio>=1.0.9
 Requires-Dist: marshmallow>=3
 Requires-Dist: aiobotocore[boto3]>=2.21.1 ; extra == "aiobotocore"
 Requires-Dist: apache-airflow-providers-apache-hive ; extra == "apache-hive"
-Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0rc1 ; extra == "cncf-kubernetes"
-Requires-Dist: apache-airflow-providers-common-messaging>=1.0.1rc1 ; extra == "common-messaging"
+Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0 ; extra == "cncf-kubernetes"
+Requires-Dist: apache-airflow-providers-common-messaging>=1.0.1 ; extra == "common-messaging"
 Requires-Dist: apache-airflow-providers-exasol ; extra == "exasol"
 Requires-Dist: apache-airflow-providers-fab ; extra == "fab"
 Requires-Dist: apache-airflow-providers-ftp ; extra == "ftp"
@@ -46,7 +45,7 @@ Requires-Dist: apache-airflow-providers-google ; extra == "google"
 Requires-Dist: apache-airflow-providers-imap ; extra == "imap"
 Requires-Dist: apache-airflow-providers-microsoft-azure ; extra == "microsoft-azure"
 Requires-Dist: apache-airflow-providers-mongo ; extra == "mongo"
-Requires-Dist: apache-airflow-providers-openlineage>=2.3.0rc1 ; extra == "openlineage"
+Requires-Dist: apache-airflow-providers-openlineage>=2.3.0 ; extra == "openlineage"
 Requires-Dist: python3-saml>=1.16.0 ; extra == "python3-saml"
 Requires-Dist: xmlsec!=1.3.15,>=1.3.14 ; extra == "python3-saml"
 Requires-Dist: lxml<5.4.0,>=5.3.2 ; extra == "python3-saml"
@@ -55,8 +54,8 @@ Requires-Dist: apache-airflow-providers-salesforce ; extra == "salesforce"
 Requires-Dist: apache-airflow-providers-ssh ; extra == "ssh"
 Requires-Dist: apache-airflow-providers-standard ; extra == "standard"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.0/changelog.html
-Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -104,7 +103,7 @@ Provides-Extra: standard
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``9.9.0``
+Release: ``9.10.0``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -117,7 +116,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/>`_.
 
 Installation
 ------------
@@ -126,7 +125,7 @@ You can install this package on top of an existing Airflow 2 installation (see `
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-amazon``
 
-The package supports the following python versions: 3.9,3.10,3.11,3.12
+The package supports the following python versions: 3.10,3.11,3.12
 
 Requirements
 ------------
@@ -185,5 +184,5 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html>`_.
 

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/README.rst

@@ -23,7 +23,7 @@
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``9.9.0``
+Release: ``9.10.0``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -36,7 +36,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/>`_.
 
 Installation
 ------------
@@ -45,7 +45,7 @@ You can install this package on top of an existing Airflow 2 installation (see `
 for the minimum Airflow version supported) via
 ``pip install apache-airflow-providers-amazon``
 
-The package supports the following python versions: 3.9,3.10,3.11,3.12
+The package supports the following python versions: 3.10,3.11,3.12
 
 Requirements
 ------------
@@ -104,4 +104,4 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html>`_.

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/docs/changelog.rst

@@ -26,6 +26,63 @@
 Changelog
 ---------
 
+9.10.0
+......
+
+Features
+~~~~~~~~
+
+* ``Add support for S3 dag bundle (#46621)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix GlueJobOperator deferred waiting (#52314)``
+* ``Handle exceptions when fetching status in GlueJobHook (#52262)``
+* ``Handle 'S3KeySensor' in 'deferrable' mode splits 'bucket_key' into individual chars (#52983)``
+* ``Pass the region_name from the GlueJobOperator / GlueJobSensor to the Trigger (#52904)``
+
+Misc
+~~~~
+
+* ``Move 'BaseHook' implementation to task SDK (#51873)``
+* ``Replace 'models.BaseOperator' to Task SDK one for Amazon Provider (#52667)``
+* ``Disable UP038 ruff rule and revert mandatory 'X | Y' in insintance checks (#52644)``
+* ``Upgrade ruff to latest version (0.12.1) (#52562)``
+* ``Replace usage of 'set_extra' with 'extra' for athena sql hook (#52340)``
+* ``Drop support for Python 3.9 (#52072)``
+* ``Replace 'models.BaseOperator' to Task SDK one for Standard Provider (#52292)``
+* ``Replace occurences of 'get_password' with 'password' to ease migration (#52333)``
+* ``Use BaseSensorOperator from task sdk in providers (#52296)``
+* ``Use base AWS classes in Glue Trigger / Sensor and implement custom waiter (#52243)``
+* ``Add Airflow 3.0+ Task SDK support to AWS Batch Executor (#52121)``
+* ``Refactor operator_extra_links property in BatchOperator (#51385)``
+* ``Remove unused batch methods from auth manager (#52883)``
+* ``Add debug logging for endpoint_url in AWS Connection (#52856)``
+* ``Remove 'MENU' from 'ResourceMethod' in auth manager (#52731)``
+* ``Remove 'MAX_XCOM_SIZE' hardcoded constant from Airflow core  (#52978)``
+* ``More robust handling of 'BaseHook.get_connection''s 'CONNECTION_NOT_FOUND' Task SDK exception (#52838)``
+* ``Move all BaseHook usages to version_compat in Amazon (#52796)``
+* ``Remove upper-binding for "python-requires" (#52980)``
+* ``Temporarily switch to use >=,< pattern instead of '~=' (#52967)``
+
+Doc-only
+~~~~~~~~
+
+* ``Clean some leftovers of Python 3.9 removal - All the rest (#52432)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Make sure all test version imports come from test_common (#52425)``
+   * ``Added additional steps to QuickSights test prerequisites (#52198)``
+   * ``examples dags: Update redshift node version dc2.large is deprecated (#52120)``
+   * ``Updating AWS systest to do connection setup using ENV (#52073)``
+   * ``Remove pytest.mark.db_test: airbyte and amazon providers where possible (#52017)``
+   * ``Introducing fixture to create 'Connections' without DB in provider tests (#51930)``
+   * ``Switch the Supervisor/task process from line-based to length-prefixed (#51699)``
+   * ``Mocked time.sleep to avoid actual sleep time (#51752)``
+   * ``Prepare release for July 2025 1st provider wave (#52727)``
+
 9.9.0
 .....
 

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/docs/executors/general.rst

@@ -142,9 +142,9 @@ which is running the Airflow scheduler process (and thus, the |executorName|
 executor.) Apache Airflow images with specific python versions can be
 downloaded from the Dockerhub registry, and filtering tags by the
 `python
-version <https://hub.docker.com/r/apache/airflow/tags?page=1&name=3.9>`__.
-For example, the tag ``latest-python3.9`` specifies that the image will
-have python 3.9 installed.
+version <https://hub.docker.com/r/apache/airflow/tags?page=1&name=3.10>`__.
+For example, the tag ``latest-python3.10`` specifies that the image will
+have python 3.10 installed.
 
 .. END BASE_IMAGE
 

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/docs/index.rst

@@ -87,7 +87,7 @@ apache-airflow-providers-amazon package
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
 
 
-Release: 9.9.0
+Release: 9.10.0
 
 Provider package
 ----------------
@@ -166,5 +166,5 @@ Downloading official packages
 You can download officially released packages and verify their checksums and signatures from the
 `Official Apache Download site <https://downloads.apache.org/airflow/providers/>`_
 
-* `The apache-airflow-providers-amazon 9.9.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.0.tar.gz.sha512>`__)
-* `The apache-airflow-providers-amazon 9.9.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.9.0-py3-none-any.whl.sha512>`__)
+* `The apache-airflow-providers-amazon 9.10.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0.tar.gz.sha512>`__)
+* `The apache-airflow-providers-amazon 9.10.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_amazon-9.10.0-py3-none-any.whl.sha512>`__)

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/provider.yaml

@@ -22,12 +22,13 @@ description: |
   Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
 
 state: ready
-source-date-epoch: 1749896002
+source-date-epoch: 1751472351
 # Note that those versions are maintained by release manager - do not update them manually
 # with the exception of case where other provider in sources has >= new provider version.
 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
 # to be done in the same PR
 versions:
+  - 9.10.0
   - 9.9.0
   - 9.8.0
   - 9.7.0

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-amazon"
-version = "9.9.0rc1"
+version = "9.10.0"
 description = "Provider package apache-airflow-providers-amazon for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -44,22 +44,21 @@ classifiers = [
     "Framework :: Apache Airflow",
     "Framework :: Apache Airflow :: Provider",
     "License :: OSI Approved :: Apache Software License",
-    "Programming Language :: Python :: 3.9",
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Topic :: System :: Monitoring",
 ]
-requires-python = "~=3.9"
+requires-python = ">=3.10"
 
 # The dependencies should be modified in place in the generated file.
 # Any change in the dependencies is preserved when the file is regenerated
 # Make sure to run ``breeze static-checks --type update-providers-dependencies --all-files``
 # After you modify the dependencies, and rebuild your Breeze CI image with ``breeze ci-image build``
 dependencies = [
-    "apache-airflow>=2.10.0rc1",
-    "apache-airflow-providers-common-compat>=1.6.1rc1",
-    "apache-airflow-providers-common-sql>=1.27.0rc1",
+    "apache-airflow>=2.10.0",
+    "apache-airflow-providers-common-compat>=1.6.1",
+    "apache-airflow-providers-common-sql>=1.27.0",
     "apache-airflow-providers-http",
     # We should update minimum version of boto3 and here regularly to avoid `pip` backtracking with the number
     # of candidates to consider. Make sure to configure boto3 version here as well as in all the tools below
@@ -90,7 +89,7 @@ dependencies = [
     "aiobotocore[boto3]>=2.21.1",
 ]
 "cncf.kubernetes" = [
-    "apache-airflow-providers-cncf-kubernetes>=7.2.0rc1",
+    "apache-airflow-providers-cncf-kubernetes>=7.2.0",
 ]
 "s3fs" = [
     "s3fs>=2023.10.0",
@@ -127,7 +126,7 @@ dependencies = [
     "apache-airflow-providers-mongo"
 ]
 "openlineage" = [
-    "apache-airflow-providers-openlineage>=2.3.0rc1"
+    "apache-airflow-providers-openlineage>=2.3.0"
 ]
 "salesforce" = [
     "apache-airflow-providers-salesforce"
@@ -139,7 +138,7 @@ dependencies = [
     "apache-airflow-providers-standard"
 ]
 "common.messaging" = [
-    "apache-airflow-providers-common-messaging>=1.0.1rc1"
+    "apache-airflow-providers-common-messaging>=1.0.1"
 ]
 
 [dependency-groups]
@@ -205,8 +204,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.0"
-"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-amazon/9.9.0/changelog.html"
+"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0"
+"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.10.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/src/airflow/providers/amazon/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "9.9.0"
+__version__ = "9.10.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.10.0"

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/src/airflow/providers/amazon/aws/auth_manager/avp/facade.py

@@ -37,6 +37,13 @@ from airflow.utils.log.logging_mixin import LoggingMixin
 
 if TYPE_CHECKING:
     from airflow.api_fastapi.auth.managers.base_auth_manager import ResourceMethod
+
+    try:
+        from airflow.api_fastapi.auth.managers.base_auth_manager import ExtendedResourceMethod
+    except ImportError:
+        from airflow.api_fastapi.auth.managers.base_auth_manager import (
+            ResourceMethod as ExtendedResourceMethod,
+        )
     from airflow.providers.amazon.aws.auth_manager.user import AwsAuthManagerUser
 
 
@@ -48,7 +55,7 @@ NB_REQUESTS_PER_BATCH = 30
 class IsAuthorizedRequest(TypedDict, total=False):
     """Represent the parameters of ``is_authorized`` method in AVP facade."""
 
-    method: ResourceMethod
+    method: ExtendedResourceMethod
     entity_type: AvpEntities
     entity_id: str | None
     context: dict | None

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/src/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py

@@ -44,10 +44,7 @@ from airflow.providers.amazon.version_compat import AIRFLOW_V_3_0_PLUS
 if TYPE_CHECKING:
     from airflow.api_fastapi.auth.managers.base_auth_manager import ResourceMethod
     from airflow.api_fastapi.auth.managers.models.batch_apis import (
-        IsAuthorizedConnectionRequest,
         IsAuthorizedDagRequest,
-        IsAuthorizedPoolRequest,
-        IsAuthorizedVariableRequest,
     )
     from airflow.api_fastapi.auth.managers.models.resource_details import (
         AccessView,
@@ -247,24 +244,6 @@ class AwsAuthManager(BaseAuthManager[AwsAuthManagerUser]):
 
         return [menu_item for menu_item in menu_items if _has_access_to_menu_item(requests[menu_item.value])]
 
-    def batch_is_authorized_connection(
-        self,
-        requests: Sequence[IsAuthorizedConnectionRequest],
-        *,
-        user: AwsAuthManagerUser,
-    ) -> bool:
-        facade_requests: Sequence[IsAuthorizedRequest] = [
-            {
-                "method": request["method"],
-                "entity_type": AvpEntities.CONNECTION,
-                "entity_id": cast("ConnectionDetails", request["details"]).conn_id
-                if request.get("details")
-                else None,
-            }
-            for request in requests
-        ]
-        return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
-
     def batch_is_authorized_dag(
         self,
         requests: Sequence[IsAuthorizedDagRequest],
@@ -288,40 +267,6 @@ class AwsAuthManager(BaseAuthManager[AwsAuthManagerUser]):
         ]
         return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
 
-    def batch_is_authorized_pool(
-        self,
-        requests: Sequence[IsAuthorizedPoolRequest],
-        *,
-        user: AwsAuthManagerUser,
-    ) -> bool:
-        facade_requests: Sequence[IsAuthorizedRequest] = [
-            {
-                "method": request["method"],
-                "entity_type": AvpEntities.POOL,
-                "entity_id": cast("PoolDetails", request["details"]).name if request.get("details") else None,
-            }
-            for request in requests
-        ]
-        return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
-
-    def batch_is_authorized_variable(
-        self,
-        requests: Sequence[IsAuthorizedVariableRequest],
-        *,
-        user: AwsAuthManagerUser,
-    ) -> bool:
-        facade_requests: Sequence[IsAuthorizedRequest] = [
-            {
-                "method": request["method"],
-                "entity_type": AvpEntities.VARIABLE,
-                "entity_id": cast("VariableDetails", request["details"]).key
-                if request.get("details")
-                else None,
-            }
-            for request in requests
-        ]
-        return self.avp_facade.batch_is_authorized(requests=facade_requests, user=user)
-
     def filter_authorized_dag_ids(
         self,
         *,

apache_airflow_providers_amazon-9.10.0/src/airflow/providers/amazon/aws/bundles/s3.py

@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import structlog
+
+from airflow.dag_processing.bundles.base import BaseDagBundle
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+
+
+class S3DagBundle(BaseDagBundle):
+    """
+    S3 DAG bundle - exposes a directory in S3 as a DAG bundle.
+
+    This allows Airflow to load DAGs directly from an S3 bucket.
+
+    :param aws_conn_id: Airflow connection ID for AWS.  Defaults to AwsBaseHook.default_conn_name.
+    :param bucket_name: The name of the S3 bucket containing the DAG files.
+    :param prefix:  Optional subdirectory within the S3 bucket where the DAGs are stored.
+                    If None, DAGs are assumed to be at the root of the bucket (Optional).
+    """
+
+    supports_versioning = False
+
+    def __init__(
+        self,
+        *,
+        aws_conn_id: str = AwsBaseHook.default_conn_name,
+        bucket_name: str,
+        prefix: str = "",
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.aws_conn_id = aws_conn_id
+        self.bucket_name = bucket_name
+        self.prefix = prefix
+        # Local path where S3 DAGs are downloaded
+        self.s3_dags_dir: Path = self.base_dir
+
+        log = structlog.get_logger(__name__)
+        self._log = log.bind(
+            bundle_name=self.name,
+            version=self.version,
+            bucket_name=self.bucket_name,
+            prefix=self.prefix,
+            aws_conn_id=self.aws_conn_id,
+        )
+        self._s3_hook: S3Hook | None = None
+
+    def _initialize(self):
+        with self.lock():
+            if not self.s3_dags_dir.exists():
+                self._log.info("Creating local DAGs directory: %s", self.s3_dags_dir)
+                os.makedirs(self.s3_dags_dir)
+
+            if not self.s3_dags_dir.is_dir():
+                raise AirflowException(f"Local DAGs path: {self.s3_dags_dir} is not a directory.")
+
+            if not self.s3_hook.check_for_bucket(bucket_name=self.bucket_name):
+                raise AirflowException(f"S3 bucket '{self.bucket_name}' does not exist.")
+
+            if self.prefix:
+                # don't check when prefix is ""
+                if not self.s3_hook.check_for_prefix(
+                    bucket_name=self.bucket_name, prefix=self.prefix, delimiter="/"
+                ):
+                    raise AirflowException(
+                        f"S3 prefix 's3://{self.bucket_name}/{self.prefix}' does not exist."
+                    )
+            self.refresh()
+
+    def initialize(self) -> None:
+        self._initialize()
+        super().initialize()
+
+    @property
+    def s3_hook(self):
+        if self._s3_hook is None:
+            try:
+                self._s3_hook: S3Hook = S3Hook(aws_conn_id=self.aws_conn_id)  # Initialize S3 hook.
+            except AirflowException as e:
+                self._log.warning("Could not create S3Hook for connection %s: %s", self.aws_conn_id, e)
+        return self._s3_hook
+
+    def __repr__(self):
+        return (
+            f"<S3DagBundle("
+            f"name={self.name!r}, "
+            f"bucket_name={self.bucket_name!r}, "
+            f"prefix={self.prefix!r}, "
+            f"version={self.version!r}"
+            f")>"
+        )
+
+    def get_current_version(self) -> str | None:
+        """Return the current version of the DAG bundle. Currently not supported."""
+        return None
+
+    @property
+    def path(self) -> Path:
+        """Return the local path to the DAG files."""
+        return self.s3_dags_dir  # Path where DAGs are downloaded.
+
+    def refresh(self) -> None:
+        """Refresh the DAG bundle by re-downloading the DAGs from S3."""
+        if self.version:
+            raise AirflowException("Refreshing a specific version is not supported")
+
+        with self.lock():
+            self._log.debug(
+                "Downloading DAGs from s3://%s/%s to %s", self.bucket_name, self.prefix, self.s3_dags_dir
+            )
+            self.s3_hook.sync_to_local_dir(
+                bucket_name=self.bucket_name,
+                s3_prefix=self.prefix,
+                local_dir=self.s3_dags_dir,
+                delete_stale=True,
+            )
+
+    def view_url(self, version: str | None = None) -> str | None:
+        """Return a URL for viewing the DAGs in S3. Currently, versioning is not supported."""
+        if self.version:
+            raise AirflowException("S3 url with version is not supported")
+
+        # https://<bucket-name>.s3.<region>.amazonaws.com/<object-key>
+        url = f"https://{self.bucket_name}.s3"
+        if self.s3_hook.region_name:
+            url += f".{self.s3_hook.region_name}"
+        url += ".amazonaws.com"
+        if self.prefix:
+            url += f"/{self.prefix}"
+
+        return url

{apache_airflow_providers_amazon-9.9.0rc1 → apache_airflow_providers_amazon-9.10.0}/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py

@@ -36,11 +36,15 @@ from airflow.providers.amazon.aws.executors.utils.exponential_backoff_retry impo
     exponential_backoff_retry,
 )
 from airflow.providers.amazon.aws.hooks.batch_client import BatchClientHook
+from airflow.providers.amazon.version_compat import AIRFLOW_V_3_0_PLUS
 from airflow.stats import Stats
 from airflow.utils import timezone
 from airflow.utils.helpers import merge_dicts
 
 if TYPE_CHECKING:
+    from sqlalchemy.orm import Session
+
+    from airflow.executors import workloads
     from airflow.models.taskinstance import TaskInstance, TaskInstanceKey
 from airflow.providers.amazon.aws.executors.batch.boto_schema import (
     BatchDescribeJobsResponseSchema,
@@ -97,6 +101,11 @@ class AwsBatchExecutor(BaseExecutor):
     # AWS only allows a maximum number of JOBs in the describe_jobs function
     DESCRIBE_JOBS_BATCH_SIZE = 99
 
+    if TYPE_CHECKING and AIRFLOW_V_3_0_PLUS:
+        # In the v3 path, we store workloads, not commands as strings.
+        # TODO: TaskSDK: move this type change into BaseExecutor
+        queued_tasks: dict[TaskInstanceKey, workloads.All]  # type: ignore[assignment]
+
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.active_workers = BatchJobCollection()
@@ -106,6 +115,30 @@ class AwsBatchExecutor(BaseExecutor):
         self.IS_BOTO_CONNECTION_HEALTHY = False
         self.submit_job_kwargs = self._load_submit_kwargs()
 
+    def queue_workload(self, workload: workloads.All, session: Session | None) -> None:
+        from airflow.executors import workloads
+
+        if not isinstance(workload, workloads.ExecuteTask):
+            raise RuntimeError(f"{type(self)} cannot handle workloads of type {type(workload)}")
+        ti = workload.ti
+        self.queued_tasks[ti.key] = workload
+
+    def _process_workloads(self, workloads: Sequence[workloads.All]) -> None:
+        from airflow.executors.workloads import ExecuteTask
+
+        # Airflow V3 version
+        for w in workloads:
+            if not isinstance(w, ExecuteTask):
+                raise RuntimeError(f"{type(self)} cannot handle workloads of type {type(w)}")
+            command = [w]
+            key = w.ti.key
+            queue = w.ti.queue
+            executor_config = w.ti.executor_config or {}
+
+            del self.queued_tasks[key]
+            self.execute_async(key=key, command=command, queue=queue, executor_config=executor_config)  # type: ignore[arg-type]
+            self.running.add(key)
+
     def check_health(self):
         """Make a test API call to check the health of the Batch Executor."""
         success_status = "succeeded."
@@ -343,6 +376,24 @@ class AwsBatchExecutor(BaseExecutor):
         if executor_config and "command" in executor_config:
             raise ValueError('Executor Config should never override "command"')
 
+        if len(command) == 1:
+            from airflow.executors.workloads import ExecuteTask
+
+            if isinstance(command[0], ExecuteTask):
+                workload = command[0]
+                ser_input = workload.model_dump_json()
+                command = [
+                    "python",
+                    "-m",
+                    "airflow.sdk.execution_time.execute_workload",
+                    "--json-string",
+                    ser_input,
+                ]
+            else:
+                raise ValueError(
+                    f"BatchExecutor doesn't know how to handle workload of type: {type(command[0])}"
+                )
+
         self.pending_jobs.append(
             BatchQueuedJob(
                 key=key,