apache-airflow-providers-amazon 8.28.0rc1__tar.gz → 8.29.0rc1__tar.gz

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: apache-airflow-providers-amazon
-Version: 8.28.0rc1
+Version: 8.29.0rc1
 Summary: Provider package apache-airflow-providers-amazon for Apache Airflow
 Keywords: airflow-provider,amazon,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -23,7 +23,6 @@ Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
 Requires-Dist: PyAthena>=3.0.10
 Requires-Dist: apache-airflow-providers-common-compat>=1.1.0rc0
-Requires-Dist: apache-airflow-providers-common-compat>=1.1.0rc0
 Requires-Dist: apache-airflow-providers-common-sql>=1.3.1rc0
 Requires-Dist: apache-airflow-providers-http
 Requires-Dist: apache-airflow>=2.8.0rc0
@@ -33,9 +32,10 @@ Requires-Dist: botocore>=1.34.90
 Requires-Dist: inflection>=0.5.1
 Requires-Dist: jmespath>=0.7.0
 Requires-Dist: jsonpath_ng>=1.5.3
+Requires-Dist: python3-saml>=1.16.0
 Requires-Dist: redshift_connector>=2.0.918
 Requires-Dist: sqlalchemy_redshift>=0.8.6
-Requires-Dist: watchtower>=3.0.0,<4
+Requires-Dist: watchtower>=3.0.0,!=3.3.0,<4
 Requires-Dist: aiobotocore[boto3]>=2.13.0 ; extra == "aiobotocore"
 Requires-Dist: apache-airflow-providers-apache-hive ; extra == "apache.hive"
 Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0rc0 ; extra == "cncf.kubernetes"
@@ -56,8 +56,8 @@ Requires-Dist: s3fs>=2023.10.0 ; extra == "s3fs"
 Requires-Dist: apache-airflow-providers-salesforce ; extra == "salesforce"
 Requires-Dist: apache-airflow-providers-ssh ; extra == "ssh"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.28.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.28.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.29.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.29.0
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
 Project-URL: Twitter, https://twitter.com/ApacheAirflow
@@ -125,7 +125,7 @@ Provides-Extra: ssh
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``8.28.0.rc1``
+Release: ``8.29.0.rc1``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -138,7 +138,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.28.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.29.0/>`_.
 
 Installation
 ------------
@@ -152,25 +152,25 @@ The package supports the following python versions: 3.8,3.9,3.10,3.11,3.12
 Requirements
 ------------
 
-==========================================  ==================
+==========================================  ======================
 PIP package                                 Version required
-==========================================  ==================
+==========================================  ======================
 ``apache-airflow``                          ``>=2.8.0``
 ``apache-airflow-providers-common-compat``  ``>=1.1.0``
 ``apache-airflow-providers-common-sql``     ``>=1.3.1``
 ``apache-airflow-providers-http``
-``apache-airflow-providers-common-compat``  ``>=1.1.0``
 ``boto3``                                   ``>=1.34.90``
 ``botocore``                                ``>=1.34.90``
 ``inflection``                              ``>=0.5.1``
-``watchtower``                              ``>=3.0.0,<4``
+``watchtower``                              ``>=3.0.0,!=3.3.0,<4``
 ``jsonpath_ng``                             ``>=1.5.3``
 ``redshift_connector``                      ``>=2.0.918``
 ``sqlalchemy_redshift``                     ``>=0.8.6``
 ``asgiref``                                 ``>=2.3.0``
 ``PyAthena``                                ``>=3.0.10``
 ``jmespath``                                ``>=0.7.0``
-==========================================  ==================
+``python3-saml``                            ``>=1.16.0``
+==========================================  ======================
 
 Cross provider package dependencies
 -----------------------------------
@@ -205,4 +205,4 @@ Dependent package
 ======================================================================================================================  ===================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.28.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.29.0/changelog.html>`_.

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/README.rst

@@ -42,7 +42,7 @@
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``8.28.0.rc1``
+Release: ``8.29.0.rc1``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -55,7 +55,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.28.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.29.0/>`_.
 
 Installation
 ------------
@@ -69,25 +69,25 @@ The package supports the following python versions: 3.8,3.9,3.10,3.11,3.12
 Requirements
 ------------
 
-==========================================  ==================
+==========================================  ======================
 PIP package                                 Version required
-==========================================  ==================
+==========================================  ======================
 ``apache-airflow``                          ``>=2.8.0``
 ``apache-airflow-providers-common-compat``  ``>=1.1.0``
 ``apache-airflow-providers-common-sql``     ``>=1.3.1``
 ``apache-airflow-providers-http``
-``apache-airflow-providers-common-compat``  ``>=1.1.0``
 ``boto3``                                   ``>=1.34.90``
 ``botocore``                                ``>=1.34.90``
 ``inflection``                              ``>=0.5.1``
-``watchtower``                              ``>=3.0.0,<4``
+``watchtower``                              ``>=3.0.0,!=3.3.0,<4``
 ``jsonpath_ng``                             ``>=1.5.3``
 ``redshift_connector``                      ``>=2.0.918``
 ``sqlalchemy_redshift``                     ``>=0.8.6``
 ``asgiref``                                 ``>=2.3.0``
 ``PyAthena``                                ``>=3.0.10``
 ``jmespath``                                ``>=0.7.0``
-==========================================  ==================
+``python3-saml``                            ``>=1.16.0``
+==========================================  ======================
 
 Cross provider package dependencies
 -----------------------------------
@@ -122,4 +122,4 @@ Dependent package
 ======================================================================================================================  ===================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.28.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.29.0/changelog.html>`_.

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "8.28.0"
+__version__ = "8.29.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.8.0"

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py

@@ -36,6 +36,7 @@ from airflow.providers.amazon.aws.auth_manager.cli.definition import (
 from airflow.providers.amazon.aws.auth_manager.security_manager.aws_security_manager_override import (
     AwsSecurityManagerOverride,
 )
+from airflow.providers.amazon.aws.auth_manager.views.auth import AwsAuthManagerAuthenticationViews
 
 try:
     from airflow.auth.managers.base_auth_manager import BaseAuthManager, ResourceMethod
@@ -423,6 +424,9 @@ class AwsAuthManager(BaseAuthManager):
             ),
         ]
 
+    def register_views(self) -> None:
+        self.appbuilder.add_view_no_menu(AwsAuthManagerAuthenticationViews())
+
     @staticmethod
     def _get_menu_item_request(resource_name: str) -> IsAuthorizedRequest:
         return {

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/auth_manager/security_manager/aws_security_manager_override.py

@@ -27,7 +27,11 @@ except ImportError:
 
 
 class AwsSecurityManagerOverride(AirflowSecurityManagerV2):
-    """The security manager override specific to AWS auth manager."""
+    """
+    The security manager override specific to AWS auth manager.
+
+    This class is only used in Airflow 2. This can be safely be removed when min Airflow version >= 3
+    """
 
     def register_views(self):
         """Register views specific to AWS auth manager."""

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/executors/batch/batch_executor.py

@@ -448,7 +448,7 @@ class AwsBatchExecutor(BaseExecutor):
                         airflow_cmd=ti.command_as_list(),
                         queue=ti.queue,
                         exec_config=ti.executor_config,
-                        attempt_number=ti.prev_attempted_tries,
+                        attempt_number=ti.try_number,
                     )
                     adopted_tis.append(ti)
 

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/executors/ecs/ecs_executor_config.py

@@ -35,6 +35,8 @@ from json import JSONDecodeError
 from airflow.configuration import conf
 from airflow.providers.amazon.aws.executors.ecs.utils import (
     CONFIG_GROUP_NAME,
+    ECS_LAUNCH_TYPE_EC2,
+    ECS_LAUNCH_TYPE_FARGATE,
     AllEcsConfigKeys,
     RunTaskKwargsConfigKeys,
     camelize_dict_keys,
@@ -56,13 +58,15 @@ def _fetch_config_values() -> dict[str, str]:
 
 
 def build_task_kwargs() -> dict:
+    all_config_keys = AllEcsConfigKeys()
     # This will put some kwargs at the root of the dictionary that do NOT belong there. However,
     # the code below expects them to be there and will rearrange them as necessary.
     task_kwargs = _fetch_config_values()
     task_kwargs.update(_fetch_templated_kwargs())
 
-    has_launch_type: bool = "launch_type" in task_kwargs
-    has_capacity_provider: bool = "capacity_provider_strategy" in task_kwargs
+    has_launch_type: bool = all_config_keys.LAUNCH_TYPE in task_kwargs
+    has_capacity_provider: bool = all_config_keys.CAPACITY_PROVIDER_STRATEGY in task_kwargs
+    is_launch_type_ec2: bool = task_kwargs.get(all_config_keys.LAUNCH_TYPE, None) == ECS_LAUNCH_TYPE_EC2
 
     if has_capacity_provider and has_launch_type:
         raise ValueError(
@@ -75,7 +79,12 @@ def build_task_kwargs() -> dict:
         # the final fallback.
         cluster = EcsHook().conn.describe_clusters(clusters=[task_kwargs["cluster"]])["clusters"][0]
         if not cluster.get("defaultCapacityProviderStrategy"):
-            task_kwargs["launch_type"] = "FARGATE"
+            task_kwargs[all_config_keys.LAUNCH_TYPE] = ECS_LAUNCH_TYPE_FARGATE
+
+    # If you're using the EC2 launch type, you should not/can not provide the platform_version. In this
+    # case we'll drop it on the floor on behalf of the user, instead of throwing an exception.
+    if is_launch_type_ec2:
+        task_kwargs.pop(all_config_keys.PLATFORM_VERSION, None)
 
     # There can only be 1 count of these containers
     task_kwargs["count"] = 1  # type: ignore
@@ -105,7 +114,7 @@ def build_task_kwargs() -> dict:
                 "awsvpcConfiguration": {
                     "subnets": str(subnets).split(",") if subnets else None,
                     "securityGroups": str(security_groups).split(",") if security_groups else None,
-                    "assignPublicIp": parse_assign_public_ip(assign_public_ip),
+                    "assignPublicIp": parse_assign_public_ip(assign_public_ip, is_launch_type_ec2),
                 }
             }
         )

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/executors/ecs/utils.py

@@ -40,6 +40,9 @@ CommandType = List[str]
 ExecutorConfigFunctionType = Callable[[CommandType], dict]
 ExecutorConfigType = Dict[str, Any]
 
+ECS_LAUNCH_TYPE_EC2 = "EC2"
+ECS_LAUNCH_TYPE_FARGATE = "FARGATE"
+
 CONFIG_GROUP_NAME = "aws_ecs_executor"
 
 CONFIG_DEFAULTS = {
@@ -247,9 +250,12 @@ def _recursive_flatten_dict(nested_dict):
     return dict(items)
 
 
-def parse_assign_public_ip(assign_public_ip):
+def parse_assign_public_ip(assign_public_ip, is_launch_type_ec2=False):
     """Convert "assign_public_ip" from True/False to ENABLE/DISABLE."""
-    return "ENABLED" if assign_public_ip == "True" else "DISABLED"
+    # If the launch type is EC2, you cannot/should not provide the assignPublicIp parameter (which is
+    # specific to Fargate)
+    if not is_launch_type_ec2:
+        return "ENABLED" if assign_public_ip == "True" else "DISABLED"
 
 
 def camelize_dict_keys(nested_dict) -> dict:

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/hooks/s3.py

@@ -238,7 +238,7 @@ class S3Hook(AwsBaseHook):
         valid_s3_virtual_hosted_format = "https://bucket-name.s3.region-code.amazonaws.com/key-name"
         format = s3url.split("//")
         if re.match(r"s3[na]?:", format[0], re.IGNORECASE):
-            parsed_url = urlsplit(s3url)
+            parsed_url = urlsplit(s3url, allow_fragments=False)
             if not parsed_url.netloc:
                 raise S3HookUriParseFailure(
                     "Please provide a bucket name using a valid format of the form: "

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py

@@ -71,13 +71,12 @@ class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
     :param base_log_folder: base folder to store logs locally
     :param log_group_arn: ARN of the Cloudwatch log group for remote log storage
         with format ``arn:aws:logs:{region name}:{account id}:log-group:{group name}``
-    :param filename_template: template for file name (local storage) or log stream name (remote)
     """
 
     trigger_should_wrap = True
 
-    def __init__(self, base_log_folder: str, log_group_arn: str, filename_template: str | None = None):
-        super().__init__(base_log_folder, filename_template)
+    def __init__(self, base_log_folder: str, log_group_arn: str, **kwargs):
+        super().__init__(base_log_folder)
         split_arn = log_group_arn.split(":")
 
         self.handler = None

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/log/s3_task_handler.py

@@ -42,10 +42,8 @@ class S3TaskHandler(FileTaskHandler, LoggingMixin):
 
     trigger_should_wrap = True
 
-    def __init__(
-        self, base_log_folder: str, s3_log_folder: str, filename_template: str | None = None, **kwargs
-    ):
-        super().__init__(base_log_folder, filename_template)
+    def __init__(self, base_log_folder: str, s3_log_folder: str, **kwargs):
+        super().__init__(base_log_folder)
         self.handler: logging.FileHandler | None = None
         self.remote_base = s3_log_folder
         self.log_relative_path = ""

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/operators/ecs.py

@@ -373,6 +373,10 @@ class EcsRunTaskOperator(EcsBaseOperator):
         When capacity_provider_strategy is specified, the launch_type parameter is omitted.
         If no capacity_provider_strategy or launch_type is specified,
         the default capacity provider strategy for the cluster is used.
+    :param volume_configurations: the volume configurations to use when using capacity provider. The name of the volume must match
+                                  the name from the task definition.
+                                  You can configure the settings like size, volume type, IOPS, throughput and others mentioned in
+                                  (https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_TaskManagedEBSVolumeConfiguration.html)
     :param group: the name of the task group associated with the task
     :param placement_constraints: an array of placement constraint objects to use for
         the task
@@ -420,6 +424,7 @@ class EcsRunTaskOperator(EcsBaseOperator):
         "overrides",
         "launch_type",
         "capacity_provider_strategy",
+        "volume_configurations",
         "group",
         "placement_constraints",
         "placement_strategy",
@@ -450,6 +455,7 @@ class EcsRunTaskOperator(EcsBaseOperator):
         overrides: dict,
         launch_type: str = "EC2",
         capacity_provider_strategy: list | None = None,
+        volume_configurations: list | None = None,
         group: str | None = None,
         placement_constraints: list | None = None,
         placement_strategy: list | None = None,
@@ -479,6 +485,7 @@ class EcsRunTaskOperator(EcsBaseOperator):
         self.overrides = overrides
         self.launch_type = launch_type
         self.capacity_provider_strategy = capacity_provider_strategy
+        self.volume_configurations = volume_configurations
         self.group = group
         self.placement_constraints = placement_constraints
         self.placement_strategy = placement_strategy
@@ -614,6 +621,8 @@ class EcsRunTaskOperator(EcsBaseOperator):
 
         if self.capacity_provider_strategy:
             run_opts["capacityProviderStrategy"] = self.capacity_provider_strategy
+            if self.volume_configurations is not None:
+                run_opts["volumeConfigurations"] = self.volume_configurations
         elif self.launch_type:
             run_opts["launchType"] = self.launch_type
         if self.platform_version is not None:

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/operators/glue_databrew.py

@@ -17,20 +17,22 @@
 # under the License.
 from __future__ import annotations
 
-from functools import cached_property
+import warnings
 from typing import TYPE_CHECKING, Any, Sequence
 
 from airflow.configuration import conf
-from airflow.models import BaseOperator
+from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarning
 from airflow.providers.amazon.aws.hooks.glue_databrew import GlueDataBrewHook
+from airflow.providers.amazon.aws.operators.base_aws import AwsBaseOperator
 from airflow.providers.amazon.aws.triggers.glue_databrew import GlueDataBrewJobCompleteTrigger
 from airflow.providers.amazon.aws.utils import validate_execute_complete_event
+from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
 
 if TYPE_CHECKING:
     from airflow.utils.context import Context
 
 
-class GlueDataBrewStartJobOperator(BaseOperator):
+class GlueDataBrewStartJobOperator(AwsBaseOperator[GlueDataBrewHook]):
     """
     Start an AWS Glue DataBrew job.
 
@@ -47,14 +49,30 @@ class GlueDataBrewStartJobOperator(BaseOperator):
     :param deferrable: If True, the operator will wait asynchronously for the job to complete.
         This implies waiting for completion. This mode requires aiobotocore module to be installed.
         (default: False)
-    :param delay: Time in seconds to wait between status checks. Default is 30.
+    :param delay: Time in seconds to wait between status checks. (Deprecated).
+    :param waiter_delay: Time in seconds to wait between status checks. Default is 30.
+    :param waiter_max_attempts: Maximum number of attempts to check for job completion. (default: 60)
     :return: dictionary with key run_id and value of the resulting job's run_id.
+
+    :param aws_conn_id: The Airflow connection used for AWS credentials.
+        If this is ``None`` or empty then the default boto3 behaviour is used. If
+        running Airflow in a distributed manner and aws_conn_id is None or
+        empty, then default boto3 configuration would be used (and must be
+        maintained on each worker node).
+    :param region_name: AWS region_name. If not specified then the default boto3 behaviour is used.
+    :param verify: Whether or not to verify SSL certificates. See:
+        https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
+    :param botocore_config: Configuration dictionary (key-values) for botocore client. See:
+        https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html
     """
 
-    template_fields: Sequence[str] = (
+    aws_hook_class = GlueDataBrewHook
+
+    template_fields: Sequence[str] = aws_template_fields(
         "job_name",
         "wait_for_completion",
-        "delay",
+        "waiter_delay",
+        "waiter_max_attempts",
         "deferrable",
     )
 
@@ -62,21 +80,25 @@ class GlueDataBrewStartJobOperator(BaseOperator):
         self,
         job_name: str,
         wait_for_completion: bool = True,
-        delay: int = 30,
+        delay: int | None = None,
+        waiter_delay: int = 30,
+        waiter_max_attempts: int = 60,
         deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
-        aws_conn_id: str | None = "aws_default",
         **kwargs,
     ):
         super().__init__(**kwargs)
         self.job_name = job_name
         self.wait_for_completion = wait_for_completion
+        self.waiter_delay = waiter_delay
+        self.waiter_max_attempts = waiter_max_attempts
         self.deferrable = deferrable
-        self.delay = delay
-        self.aws_conn_id = aws_conn_id
-
-    @cached_property
-    def hook(self) -> GlueDataBrewHook:
-        return GlueDataBrewHook(aws_conn_id=self.aws_conn_id)
+        if delay is not None:
+            warnings.warn(
+                "please use `waiter_delay` instead of delay.",
+                AirflowProviderDeprecationWarning,
+                stacklevel=2,
+            )
+            self.waiter_delay = delay
 
     def execute(self, context: Context):
         job = self.hook.conn.start_job_run(Name=self.job_name)
@@ -88,7 +110,14 @@ class GlueDataBrewStartJobOperator(BaseOperator):
             self.log.info("Deferring job %s with run_id %s", self.job_name, run_id)
             self.defer(
                 trigger=GlueDataBrewJobCompleteTrigger(
-                    aws_conn_id=self.aws_conn_id, job_name=self.job_name, run_id=run_id, delay=self.delay
+                    job_name=self.job_name,
+                    run_id=run_id,
+                    waiter_delay=self.waiter_delay,
+                    waiter_max_attempts=self.waiter_max_attempts,
+                    aws_conn_id=self.aws_conn_id,
+                    region_name=self.region_name,
+                    verify=self.verify,
+                    botocore_config=self.botocore_config,
                 ),
                 method_name="execute_complete",
             )
@@ -97,7 +126,12 @@ class GlueDataBrewStartJobOperator(BaseOperator):
             self.log.info(
                 "Waiting for AWS Glue DataBrew Job: %s. Run Id: %s to complete.", self.job_name, run_id
             )
-            status = self.hook.job_completion(job_name=self.job_name, delay=self.delay, run_id=run_id)
+            status = self.hook.job_completion(
+                job_name=self.job_name,
+                delay=self.waiter_delay,
+                run_id=run_id,
+                max_attempts=self.waiter_max_attempts,
+            )
             self.log.info("Glue DataBrew Job: %s status: %s", self.job_name, status)
 
         return {"run_id": run_id}
@@ -105,6 +139,9 @@ class GlueDataBrewStartJobOperator(BaseOperator):
     def execute_complete(self, context: Context, event: dict[str, Any] | None = None) -> dict[str, str]:
         event = validate_execute_complete_event(event)
 
+        if event["status"] != "success":
+            raise AirflowException("Error while running AWS Glue DataBrew job: %s", event)
+
         run_id = event.get("run_id", "")
         status = event.get("status", "")
 

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/sensors/batch.py

@@ -23,7 +23,7 @@ from typing import TYPE_CHECKING, Any, Sequence
 from deprecated import deprecated
 
 from airflow.configuration import conf
-from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarning, AirflowSkipException
+from airflow.exceptions import AirflowException, AirflowProviderDeprecationWarning
 from airflow.providers.amazon.aws.hooks.batch_client import BatchClientHook
 from airflow.providers.amazon.aws.triggers.batch import BatchJobTrigger
 from airflow.sensors.base import BaseSensorOperator
@@ -265,6 +265,4 @@ class BatchJobQueueSensor(BaseSensorOperator):
             return False
 
         message = f"AWS Batch job queue failed. AWS Batch job queue status: {status}"
-        if self.soft_fail:
-            raise AirflowSkipException(message)
         raise AirflowException(message)

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/sensors/ecs.py

@@ -35,7 +35,7 @@ if TYPE_CHECKING:
     from airflow.utils.context import Context
 
 
-def _check_failed(current_state, target_state, failure_states, soft_fail: bool) -> None:
+def _check_failed(current_state, target_state, failure_states) -> None:
     if (current_state != target_state) and (current_state in failure_states):
         raise AirflowException(
             f"Terminal state reached. Current state: {current_state}, Expected state: {target_state}"
@@ -86,7 +86,7 @@ class EcsClusterStateSensor(EcsBaseSensor):
         cluster_state = EcsClusterStates(self.hook.get_cluster_state(cluster_name=self.cluster_name))
 
         self.log.info("Cluster state: %s, waiting for: %s", cluster_state, self.target_state)
-        _check_failed(cluster_state, self.target_state, self.failure_states, self.soft_fail)
+        _check_failed(cluster_state, self.target_state, self.failure_states)
 
         return cluster_state == self.target_state
 
@@ -132,7 +132,7 @@ class EcsTaskDefinitionStateSensor(EcsBaseSensor):
         )
 
         self.log.info("Task Definition state: %s, waiting for: %s", task_definition_state, self.target_state)
-        _check_failed(task_definition_state, self.target_state, [self.failure_states], self.soft_fail)
+        _check_failed(task_definition_state, self.target_state, [self.failure_states])
         return task_definition_state == self.target_state
 
 
@@ -172,5 +172,5 @@ class EcsTaskStateSensor(EcsBaseSensor):
         task_state = EcsTaskStates(self.hook.get_task_state(cluster=self.cluster, task=self.task))
 
         self.log.info("Task state: %s, waiting for: %s", task_state, self.target_state)
-        _check_failed(task_state, self.target_state, self.failure_states, self.soft_fail)
+        _check_failed(task_state, self.target_state, self.failure_states)
         return task_state == self.target_state

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/sensors/glue.py

@@ -21,7 +21,7 @@ from functools import cached_property
 from typing import TYPE_CHECKING, Any, Sequence
 
 from airflow.configuration import conf
-from airflow.exceptions import AirflowException, AirflowSkipException
+from airflow.exceptions import AirflowException
 from airflow.providers.amazon.aws.hooks.glue import GlueDataQualityHook, GlueJobHook
 from airflow.providers.amazon.aws.sensors.base_aws import AwsBaseSensor
 from airflow.providers.amazon.aws.triggers.glue import (
@@ -177,8 +177,6 @@ class GlueDataQualityRuleSetEvaluationRunSensor(AwsBaseSensor[GlueDataQualityHoo
 
         if event["status"] != "success":
             message = f"Error: AWS Glue data quality ruleset evaluation run: {event}"
-            if self.soft_fail:
-                raise AirflowSkipException(message)
             raise AirflowException(message)
 
         self.hook.validate_evaluation_run_results(
@@ -300,8 +298,6 @@ class GlueDataQualityRuleRecommendationRunSensor(AwsBaseSensor[GlueDataQualityHo
 
         if event["status"] != "success":
             message = f"Error: AWS Glue data quality recommendation run: {event}"
-            if self.soft_fail:
-                raise AirflowSkipException(message)
             raise AirflowException(message)
 
         if self.show_results:

{apache_airflow_providers_amazon-8.28.0rc1 → apache_airflow_providers_amazon-8.29.0rc1}/airflow/providers/amazon/aws/transfers/redshift_to_s3.py

@@ -44,11 +44,12 @@ class RedshiftToS3Operator(BaseOperator):
     :param s3_bucket: reference to a specific S3 bucket
     :param s3_key: reference to a specific S3 key. If ``table_as_file_name`` is set
         to False, this param must include the desired file name
-    :param schema: reference to a specific schema in redshift database
-        Applicable when ``table`` param provided.
-    :param table: reference to a specific table in redshift database
-        Used when ``select_query`` param not provided.
-    :param select_query: custom select query to fetch data from redshift database
+    :param schema: reference to a specific schema in redshift database,
+        used when ``table`` param provided and ``select_query`` param not provided
+    :param table: reference to a specific table in redshift database,
+        used when ``schema`` param provided and ``select_query`` param not provided
+    :param select_query: custom select query to fetch data from redshift database,
+        has precedence over default query `SELECT * FROM ``schema``.``table``
     :param redshift_conn_id: reference to a specific redshift database
     :param aws_conn_id: reference to a specific S3 connection
         If the AWS connection contains 'aws_iam_role' in ``extras``
@@ -138,12 +139,17 @@ class RedshiftToS3Operator(BaseOperator):
                     {unload_options};
         """
 
+    @property
+    def default_select_query(self) -> str | None:
+        if self.schema and self.table:
+            return f"SELECT * FROM {self.schema}.{self.table}"
+        return None
+
     def execute(self, context: Context) -> None:
         if self.table and self.table_as_file_name:
             self.s3_key = f"{self.s3_key}/{self.table}_"
 
-        if self.schema and self.table:
-            self.select_query = f"SELECT * FROM {self.schema}.{self.table}"
+        self.select_query = self.select_query or self.default_select_query
 
         if self.select_query is None:
             raise ValueError(