apache-airflow-providers-amazon 8.26.0rc2__tar.gz → 8.27.0rc1__tar.gz

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: apache-airflow-providers-amazon
-Version: 8.26.0rc2
+Version: 8.27.0rc1
 Summary: Provider package apache-airflow-providers-amazon for Apache Airflow
 Keywords: airflow-provider,amazon,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -22,6 +22,8 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
 Requires-Dist: PyAthena>=3.0.10
+Requires-Dist: apache-airflow-providers-common-compat>=1.1.0rc0
+Requires-Dist: apache-airflow-providers-common-compat>=1.1.0rc0
 Requires-Dist: apache-airflow-providers-common-sql>=1.3.1rc0
 Requires-Dist: apache-airflow-providers-http
 Requires-Dist: apache-airflow>=2.7.0rc0
@@ -37,6 +39,7 @@ Requires-Dist: watchtower>=3.0.0,<4
 Requires-Dist: aiobotocore[boto3]>=2.13.0 ; extra == "aiobotocore"
 Requires-Dist: apache-airflow-providers-apache-hive ; extra == "apache.hive"
 Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0rc0 ; extra == "cncf.kubernetes"
+Requires-Dist: apache-airflow-providers-common-compat ; extra == "common.compat"
 Requires-Dist: apache-airflow-providers-common-sql ; extra == "common.sql"
 Requires-Dist: apache-airflow-providers-exasol ; extra == "exasol"
 Requires-Dist: apache-airflow-providers-ftp ; extra == "ftp"
@@ -53,8 +56,8 @@ Requires-Dist: s3fs>=2023.10.0 ; extra == "s3fs"
 Requires-Dist: apache-airflow-providers-salesforce ; extra == "salesforce"
 Requires-Dist: apache-airflow-providers-ssh ; extra == "ssh"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.26.0/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.26.0
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.27.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.27.0
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
 Project-URL: Twitter, https://twitter.com/ApacheAirflow
@@ -62,6 +65,7 @@ Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
 Provides-Extra: aiobotocore
 Provides-Extra: apache.hive
 Provides-Extra: cncf.kubernetes
+Provides-Extra: common.compat
 Provides-Extra: common.sql
 Provides-Extra: exasol
 Provides-Extra: ftp
@@ -121,7 +125,7 @@ Provides-Extra: ssh
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``8.26.0.rc2``
+Release: ``8.27.0.rc1``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -134,7 +138,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.26.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.27.0/>`_.
 
 Installation
 ------------
@@ -148,23 +152,25 @@ The package supports the following python versions: 3.8,3.9,3.10,3.11,3.12
 Requirements
 ------------
 
-=======================================  ==================
-PIP package                              Version required
-=======================================  ==================
-``apache-airflow``                       ``>=2.7.0``
-``apache-airflow-providers-common-sql``  ``>=1.3.1``
+==========================================  ==================
+PIP package                                 Version required
+==========================================  ==================
+``apache-airflow``                          ``>=2.7.0``
+``apache-airflow-providers-common-compat``  ``>=1.1.0``
+``apache-airflow-providers-common-sql``     ``>=1.3.1``
 ``apache-airflow-providers-http``
-``boto3``                                ``>=1.34.90``
-``botocore``                             ``>=1.34.90``
-``inflection``                           ``>=0.5.1``
-``watchtower``                           ``>=3.0.0,<4``
-``jsonpath_ng``                          ``>=1.5.3``
-``redshift_connector``                   ``>=2.0.918``
-``sqlalchemy_redshift``                  ``>=0.8.6``
-``asgiref``                              ``>=2.3.0``
-``PyAthena``                             ``>=3.0.10``
-``jmespath``                             ``>=0.7.0``
-=======================================  ==================
+``apache-airflow-providers-common-compat``  ``>=1.1.0``
+``boto3``                                   ``>=1.34.90``
+``botocore``                                ``>=1.34.90``
+``inflection``                              ``>=0.5.1``
+``watchtower``                              ``>=3.0.0,<4``
+``jsonpath_ng``                             ``>=1.5.3``
+``redshift_connector``                      ``>=2.0.918``
+``sqlalchemy_redshift``                     ``>=0.8.6``
+``asgiref``                                 ``>=2.3.0``
+``PyAthena``                                ``>=3.0.10``
+``jmespath``                                ``>=0.7.0``
+==========================================  ==================
 
 Cross provider package dependencies
 -----------------------------------
@@ -184,6 +190,7 @@ Dependent package
 ======================================================================================================================  ===================
 `apache-airflow-providers-apache-hive <https://airflow.apache.org/docs/apache-airflow-providers-apache-hive>`_          ``apache.hive``
 `apache-airflow-providers-cncf-kubernetes <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes>`_  ``cncf.kubernetes``
+`apache-airflow-providers-common-compat <https://airflow.apache.org/docs/apache-airflow-providers-common-compat>`_      ``common.compat``
 `apache-airflow-providers-common-sql <https://airflow.apache.org/docs/apache-airflow-providers-common-sql>`_            ``common.sql``
 `apache-airflow-providers-exasol <https://airflow.apache.org/docs/apache-airflow-providers-exasol>`_                    ``exasol``
 `apache-airflow-providers-ftp <https://airflow.apache.org/docs/apache-airflow-providers-ftp>`_                          ``ftp``
@@ -198,4 +205,4 @@ Dependent package
 ======================================================================================================================  ===================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.26.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.27.0/changelog.html>`_.

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/README.rst

@@ -42,7 +42,7 @@
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``8.26.0.rc2``
+Release: ``8.27.0.rc1``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -55,7 +55,7 @@ This is a provider package for ``amazon`` provider. All classes for this provide
 are in ``airflow.providers.amazon`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.26.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.27.0/>`_.
 
 Installation
 ------------
@@ -69,23 +69,25 @@ The package supports the following python versions: 3.8,3.9,3.10,3.11,3.12
 Requirements
 ------------
 
-=======================================  ==================
-PIP package                              Version required
-=======================================  ==================
-``apache-airflow``                       ``>=2.7.0``
-``apache-airflow-providers-common-sql``  ``>=1.3.1``
+==========================================  ==================
+PIP package                                 Version required
+==========================================  ==================
+``apache-airflow``                          ``>=2.7.0``
+``apache-airflow-providers-common-compat``  ``>=1.1.0``
+``apache-airflow-providers-common-sql``     ``>=1.3.1``
 ``apache-airflow-providers-http``
-``boto3``                                ``>=1.34.90``
-``botocore``                             ``>=1.34.90``
-``inflection``                           ``>=0.5.1``
-``watchtower``                           ``>=3.0.0,<4``
-``jsonpath_ng``                          ``>=1.5.3``
-``redshift_connector``                   ``>=2.0.918``
-``sqlalchemy_redshift``                  ``>=0.8.6``
-``asgiref``                              ``>=2.3.0``
-``PyAthena``                             ``>=3.0.10``
-``jmespath``                             ``>=0.7.0``
-=======================================  ==================
+``apache-airflow-providers-common-compat``  ``>=1.1.0``
+``boto3``                                   ``>=1.34.90``
+``botocore``                                ``>=1.34.90``
+``inflection``                              ``>=0.5.1``
+``watchtower``                              ``>=3.0.0,<4``
+``jsonpath_ng``                             ``>=1.5.3``
+``redshift_connector``                      ``>=2.0.918``
+``sqlalchemy_redshift``                     ``>=0.8.6``
+``asgiref``                                 ``>=2.3.0``
+``PyAthena``                                ``>=3.0.10``
+``jmespath``                                ``>=0.7.0``
+==========================================  ==================
 
 Cross provider package dependencies
 -----------------------------------
@@ -105,6 +107,7 @@ Dependent package
 ======================================================================================================================  ===================
 `apache-airflow-providers-apache-hive <https://airflow.apache.org/docs/apache-airflow-providers-apache-hive>`_          ``apache.hive``
 `apache-airflow-providers-cncf-kubernetes <https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes>`_  ``cncf.kubernetes``
+`apache-airflow-providers-common-compat <https://airflow.apache.org/docs/apache-airflow-providers-common-compat>`_      ``common.compat``
 `apache-airflow-providers-common-sql <https://airflow.apache.org/docs/apache-airflow-providers-common-sql>`_            ``common.sql``
 `apache-airflow-providers-exasol <https://airflow.apache.org/docs/apache-airflow-providers-exasol>`_                    ``exasol``
 `apache-airflow-providers-ftp <https://airflow.apache.org/docs/apache-airflow-providers-ftp>`_                          ``ftp``
@@ -119,4 +122,4 @@ Dependent package
 ======================================================================================================================  ===================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.26.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.27.0/changelog.html>`_.

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/airflow/providers/amazon/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "8.26.0"
+__version__ = "8.27.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.7.0"

apache_airflow_providers_amazon-8.27.0rc1/airflow/providers/amazon/aws/datasets/s3.py

@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from airflow.datasets import Dataset
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+
+if TYPE_CHECKING:
+    from urllib.parse import SplitResult
+
+    from airflow.providers.common.compat.openlineage.facet import Dataset as OpenLineageDataset
+
+
+def create_dataset(*, bucket: str, key: str, extra=None) -> Dataset:
+    return Dataset(uri=f"s3://{bucket}/{key}", extra=extra)
+
+
+def sanitize_uri(uri: SplitResult) -> SplitResult:
+    if not uri.netloc:
+        raise ValueError("URI format s3:// must contain a bucket name")
+    return uri
+
+
+def convert_dataset_to_openlineage(dataset: Dataset, lineage_context) -> OpenLineageDataset:
+    """Translate Dataset with valid AIP-60 uri to OpenLineage with assistance from the hook."""
+    from airflow.providers.common.compat.openlineage.facet import Dataset as OpenLineageDataset
+
+    bucket, key = S3Hook.parse_s3_url(dataset.uri)
+    return OpenLineageDataset(namespace=f"s3://{bucket}", name=key if key else "/")

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/airflow/providers/amazon/aws/executors/batch/batch_executor.py

@@ -19,10 +19,9 @@
 
 from __future__ import annotations
 
-import contextlib
-import logging
 import time
 from collections import deque
+from contextlib import suppress
 from copy import deepcopy
 from typing import TYPE_CHECKING, Any, Dict, List, Sequence
 
@@ -292,12 +291,19 @@ class AwsBatchExecutor(BaseExecutor):
 
             if failure_reason:
                 if attempt_number >= int(self.__class__.MAX_SUBMIT_JOB_ATTEMPTS):
-                    self.send_message_to_task_logs(
-                        logging.ERROR,
-                        "This job has been unsuccessfully attempted too many times (%s). Dropping the task. Reason: %s",
+                    self.log.error(
+                        (
+                            "This job has been unsuccessfully attempted too many times (%s). "
+                            "Dropping the task. Reason: %s"
+                        ),
                         attempt_number,
                         failure_reason,
-                        ti=key,
+                    )
+                    self.log_task_event(
+                        event="batch job submit failure",
+                        extra=f"This job has been unsuccessfully attempted too many times ({attempt_number}). "
+                        f"Dropping the task. Reason: {failure_reason}",
+                        ti_key=key,
                     )
                     self.fail(key=key)
                 else:
@@ -317,7 +323,7 @@ class AwsBatchExecutor(BaseExecutor):
                     exec_config=exec_config,
                     attempt_number=attempt_number,
                 )
-                with contextlib.suppress(AttributeError):
+                with suppress(AttributeError):
                     # TODO: Remove this when min_airflow_version is 2.10.0 or higher in Amazon provider.
                     # running_state is added in Airflow 2.10 and only needed to support task adoption
                     # (an optional executor feature).
@@ -458,10 +464,11 @@ class AwsBatchExecutor(BaseExecutor):
             not_adopted_tis = [ti for ti in tis if ti not in adopted_tis]
             return not_adopted_tis
 
-    def send_message_to_task_logs(self, level: int, msg: str, *args, ti: TaskInstance | TaskInstanceKey):
+    def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
         # TODO: remove this method when min_airflow_version is set to higher than 2.10.0
-        try:
-            super().send_message_to_task_logs(level, msg, *args, ti=ti)
-        except AttributeError:
-            # ``send_message_to_task_logs`` is added in 2.10.0
-            self.log.error(msg, *args)
+        with suppress(AttributeError):
+            super().log_task_event(
+                event=event,
+                extra=extra,
+                ti_key=ti_key,
+            )

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py

@@ -23,9 +23,9 @@ Each Airflow task gets delegated out to an Amazon ECS Task.
 
 from __future__ import annotations
 
-import logging
 import time
 from collections import defaultdict, deque
+from contextlib import suppress
 from copy import deepcopy
 from typing import TYPE_CHECKING, Sequence
 
@@ -385,18 +385,28 @@ class AwsEcsExecutor(BaseExecutor):
                     )
                     self.pending_tasks.append(ecs_task)
                 else:
-                    self.send_message_to_task_logs(
-                        logging.ERROR,
+                    reasons_str = ", ".join(failure_reasons)
+                    self.log.error(
                         "ECS task %s has failed a maximum of %s times. Marking as failed. Reasons: %s",
                         task_key,
                         attempt_number,
-                        ", ".join(failure_reasons),
-                        ti=task_key,
+                        reasons_str,
+                    )
+                    self.log_task_event(
+                        event="ecs task submit failure",
+                        ti_key=task_key,
+                        extra=(
+                            f"Task could not be queued after {attempt_number} attempts. "
+                            f"Marking as failed. Reasons: {reasons_str}"
+                        ),
                     )
                     self.fail(task_key)
             elif not run_task_response["tasks"]:
-                self.send_message_to_task_logs(
-                    logging.ERROR, "ECS RunTask Response: %s", run_task_response, ti=task_key
+                self.log.error("ECS RunTask Response: %s", run_task_response)
+                self.log_task_event(
+                    event="ecs task submit failure",
+                    extra=f"ECS RunTask Response: {run_task_response}",
+                    ti_key=task_key,
                 )
                 raise EcsExecutorException(
                     "No failures and no ECS tasks provided in response. This should never happen."
@@ -543,10 +553,11 @@ class AwsEcsExecutor(BaseExecutor):
             not_adopted_tis = [ti for ti in tis if ti not in adopted_tis]
             return not_adopted_tis
 
-    def send_message_to_task_logs(self, level: int, msg: str, *args, ti: TaskInstance | TaskInstanceKey):
+    def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
         # TODO: remove this method when min_airflow_version is set to higher than 2.10.0
-        try:
-            super().send_message_to_task_logs(level, msg, *args, ti=ti)
-        except AttributeError:
-            # ``send_message_to_task_logs`` is added in 2.10.0
-            self.log.error(msg, *args)
+        with suppress(AttributeError):
+            super().log_task_event(
+                event=event,
+                extra=extra,
+                ti_key=ti_key,
+            )

apache_airflow_providers_amazon-8.27.0rc1/airflow/providers/amazon/aws/hooks/kinesis_analytics.py

@@ -0,0 +1,65 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+
+
+class KinesisAnalyticsV2Hook(AwsBaseHook):
+    """
+    Interact with Amazon Kinesis Analytics V2.
+
+    Provide thin wrapper around :external+boto3:py:class:`boto3.client("kinesisanalyticsv2") <KinesisAnalyticsV2.Client>`.
+
+    Additional arguments (such as ``aws_conn_id``) may be specified and
+    are passed down to the underlying AwsBaseHook.
+
+    .. seealso::
+        - :class:`airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
+    """
+
+    APPLICATION_START_INTERMEDIATE_STATES: tuple[str, ...] = ("STARTING", "UPDATING", "AUTOSCALING")
+    APPLICATION_START_FAILURE_STATES: tuple[str, ...] = (
+        "DELETING",
+        "STOPPING",
+        "READY",
+        "FORCE_STOPPING",
+        "ROLLING_BACK",
+        "MAINTENANCE",
+        "ROLLED_BACK",
+    )
+    APPLICATION_START_SUCCESS_STATES: tuple[str, ...] = ("RUNNING",)
+
+    APPLICATION_STOP_INTERMEDIATE_STATES: tuple[str, ...] = (
+        "STARTING",
+        "UPDATING",
+        "AUTOSCALING",
+        "RUNNING",
+        "STOPPING",
+        "FORCE_STOPPING",
+    )
+    APPLICATION_STOP_FAILURE_STATES: tuple[str, ...] = (
+        "DELETING",
+        "ROLLING_BACK",
+        "MAINTENANCE",
+        "ROLLED_BACK",
+    )
+    APPLICATION_STOP_SUCCESS_STATES: tuple[str, ...] = ("READY",)
+
+    def __init__(self, *args, **kwargs) -> None:
+        kwargs["client_type"] = "kinesisanalyticsv2"
+        super().__init__(*args, **kwargs)

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/airflow/providers/amazon/aws/hooks/rds.py

@@ -259,7 +259,7 @@ class RdsHook(AwsGenericHook["RDSClient"]):
             return self.get_db_instance_state(db_instance_id)
 
         target_state = target_state.lower()
-        if target_state in ("available", "deleted"):
+        if target_state in ("available", "deleted", "stopped"):
             waiter = self.conn.get_waiter(f"db_instance_{target_state}")  # type: ignore
             wait(
                 waiter=waiter,
@@ -272,7 +272,7 @@ class RdsHook(AwsGenericHook["RDSClient"]):
             )
         else:
             self._wait_for_state(poke, target_state, check_interval, max_attempts)
-            self.log.info("DB cluster snapshot '%s' reached the '%s' state", db_instance_id, target_state)
+            self.log.info("DB cluster '%s' reached the '%s' state", db_instance_id, target_state)
 
     def get_db_cluster_state(self, db_cluster_id: str) -> str:
         """
@@ -310,7 +310,7 @@ class RdsHook(AwsGenericHook["RDSClient"]):
             return self.get_db_cluster_state(db_cluster_id)
 
         target_state = target_state.lower()
-        if target_state in ("available", "deleted"):
+        if target_state in ("available", "deleted", "stopped"):
             waiter = self.conn.get_waiter(f"db_cluster_{target_state}")  # type: ignore
             waiter.wait(
                 DBClusterIdentifier=db_cluster_id,

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/airflow/providers/amazon/aws/hooks/s3.py

@@ -41,6 +41,8 @@ from typing import TYPE_CHECKING, Any, AsyncIterator, Callable
 from urllib.parse import urlsplit
 from uuid import uuid4
 
+from airflow.providers.common.compat.lineage.hook import get_hook_lineage_collector
+
 if TYPE_CHECKING:
     from mypy_boto3_s3.service_resource import Bucket as S3Bucket, Object as S3ResourceObject
 
@@ -1111,6 +1113,12 @@ class S3Hook(AwsBaseHook):
 
         client = self.get_conn()
         client.upload_file(filename, bucket_name, key, ExtraArgs=extra_args, Config=self.transfer_config)
+        get_hook_lineage_collector().add_input_dataset(
+            context=self, scheme="file", dataset_kwargs={"path": filename}
+        )
+        get_hook_lineage_collector().add_output_dataset(
+            context=self, scheme="s3", dataset_kwargs={"bucket": bucket_name, "key": key}
+        )
 
     @unify_bucket_name_and_key
     @provide_bucket_name
@@ -1251,6 +1259,10 @@ class S3Hook(AwsBaseHook):
             ExtraArgs=extra_args,
             Config=self.transfer_config,
         )
+        # No input because file_obj can be anything - handle in calling function if possible
+        get_hook_lineage_collector().add_output_dataset(
+            context=self, scheme="s3", dataset_kwargs={"bucket": bucket_name, "key": key}
+        )
 
     def copy_object(
         self,
@@ -1306,6 +1318,12 @@ class S3Hook(AwsBaseHook):
         response = self.get_conn().copy_object(
             Bucket=dest_bucket_name, Key=dest_bucket_key, CopySource=copy_source, **kwargs
         )
+        get_hook_lineage_collector().add_input_dataset(
+            context=self, scheme="s3", dataset_kwargs={"bucket": source_bucket_name, "key": source_bucket_key}
+        )
+        get_hook_lineage_collector().add_output_dataset(
+            context=self, scheme="s3", dataset_kwargs={"bucket": dest_bucket_name, "key": dest_bucket_key}
+        )
         return response
 
     @provide_bucket_name
@@ -1425,6 +1443,11 @@ class S3Hook(AwsBaseHook):
 
             file_path.parent.mkdir(exist_ok=True, parents=True)
 
+            get_hook_lineage_collector().add_output_dataset(
+                context=self,
+                scheme="file",
+                dataset_kwargs={"path": file_path if file_path.is_absolute() else file_path.absolute()},
+            )
             file = open(file_path, "wb")
         else:
             file = NamedTemporaryFile(dir=local_path, prefix="airflow_tmp_", delete=False)  # type: ignore
@@ -1435,7 +1458,9 @@ class S3Hook(AwsBaseHook):
                 ExtraArgs=self.extra_args,
                 Config=self.transfer_config,
             )
-
+        get_hook_lineage_collector().add_input_dataset(
+            context=self, scheme="s3", dataset_kwargs={"bucket": bucket_name, "key": key}
+        )
         return file.name
 
     def generate_presigned_url(

{apache_airflow_providers_amazon-8.26.0rc2 → apache_airflow_providers_amazon-8.27.0rc1}/airflow/providers/amazon/aws/hooks/step_function.py

@@ -18,6 +18,7 @@ from __future__ import annotations
 
 import json
 
+from airflow.exceptions import AirflowFailException
 from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
 
 
@@ -43,6 +44,7 @@ class StepFunctionHook(AwsBaseHook):
         state_machine_arn: str,
         name: str | None = None,
         state_machine_input: dict | str | None = None,
+        is_redrive_execution: bool = False,
     ) -> str:
         """
         Start Execution of the State Machine.
@@ -51,10 +53,26 @@ class StepFunctionHook(AwsBaseHook):
             - :external+boto3:py:meth:`SFN.Client.start_execution`
 
         :param state_machine_arn: AWS Step Function State Machine ARN.
+        :param is_redrive_execution: Restarts unsuccessful executions of Standard workflows that did not
+            complete successfully in the last 14 days.
         :param name: The name of the execution.
         :param state_machine_input: JSON data input to pass to the State Machine.
         :return: Execution ARN.
         """
+        if is_redrive_execution:
+            if not name:
+                raise AirflowFailException(
+                    "Execution name is required to start RedriveExecution for %s.", state_machine_arn
+                )
+            elements = state_machine_arn.split(":stateMachine:")
+            execution_arn = f"{elements[0]}:execution:{elements[1]}:{name}"
+            self.conn.redrive_execution(executionArn=execution_arn)
+            self.log.info(
+                "Successfully started RedriveExecution for Step Function State Machine: %s.",
+                state_machine_arn,
+            )
+            return execution_arn
+
         execution_args = {"stateMachineArn": state_machine_arn}
         if name is not None:
             execution_args["name"] = name