apache-airflow-providers-amazon 8.24.0__tar.gz → 8.24.0rc1__tar.gz

{apache_airflow_providers_amazon-8.24.0 → apache_airflow_providers_amazon-8.24.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: apache-airflow-providers-amazon
-Version: 8.24.0
+Version: 8.24.0rc1
 Summary: Provider package apache-airflow-providers-amazon for Apache Airflow
 Keywords: airflow-provider,amazon,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -22,21 +22,21 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
 Requires-Dist: PyAthena>=3.0.10
-Requires-Dist: apache-airflow-providers-common-sql>=1.3.1
+Requires-Dist: apache-airflow-providers-common-sql>=1.3.1rc0
 Requires-Dist: apache-airflow-providers-http
-Requires-Dist: apache-airflow>=2.7.0
-Requires-Dist: asgiref>=2.3.0
-Requires-Dist: boto3>=1.34.90
-Requires-Dist: botocore>=1.34.90
+Requires-Dist: apache-airflow>=2.7.0rc0
+Requires-Dist: asgiref
+Requires-Dist: boto3>=1.33.0
+Requires-Dist: botocore>=1.33.0
 Requires-Dist: inflection>=0.5.1
-Requires-Dist: jmespath>=0.7.0
+Requires-Dist: jmespath
 Requires-Dist: jsonpath_ng>=1.5.3
 Requires-Dist: redshift_connector>=2.0.918
 Requires-Dist: sqlalchemy_redshift>=0.8.6
-Requires-Dist: watchtower>=3.0.0,<4
-Requires-Dist: aiobotocore[boto3]>=2.13.0 ; extra == "aiobotocore"
+Requires-Dist: watchtower>=2.0.1,<4
+Requires-Dist: aiobotocore[boto3]>=2.5.3 ; extra == "aiobotocore"
 Requires-Dist: apache-airflow-providers-apache-hive ; extra == "apache.hive"
-Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0 ; extra == "cncf.kubernetes"
+Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0rc0 ; extra == "cncf.kubernetes"
 Requires-Dist: apache-airflow-providers-common-sql ; extra == "common.sql"
 Requires-Dist: apache-airflow-providers-exasol ; extra == "exasol"
 Requires-Dist: apache-airflow-providers-ftp ; extra == "ftp"
@@ -46,8 +46,7 @@ Requires-Dist: apache-airflow-providers-imap ; extra == "imap"
 Requires-Dist: apache-airflow-providers-microsoft-azure ; extra == "microsoft.azure"
 Requires-Dist: apache-airflow-providers-mongo ; extra == "mongo"
 Requires-Dist: apache-airflow-providers-openlineage ; extra == "openlineage"
-Requires-Dist: pandas>=1.5.3,<2.2 ; extra == "pandas" and (python_version<"3.12")
-Requires-Dist: pandas>=2.1.1,<2.2 ; extra == "pandas" and (python_version>="3.12")
+Requires-Dist: pandas>=1.2.5,<2.2 ; extra == "pandas"
 Requires-Dist: python3-saml>=1.16.0 ; extra == "python3-saml"
 Requires-Dist: s3fs>=2023.10.0 ; extra == "s3fs"
 Requires-Dist: apache-airflow-providers-salesforce ; extra == "salesforce"
@@ -121,7 +120,7 @@ Provides-Extra: ssh
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``8.24.0``
+Release: ``8.24.0.rc1``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -154,16 +153,16 @@ PIP package                              Version required
 ``apache-airflow``                       ``>=2.7.0``
 ``apache-airflow-providers-common-sql``  ``>=1.3.1``
 ``apache-airflow-providers-http``
-``boto3``                                ``>=1.34.90``
-``botocore``                             ``>=1.34.90``
+``boto3``                                ``>=1.33.0``
+``botocore``                             ``>=1.33.0``
 ``inflection``                           ``>=0.5.1``
-``watchtower``                           ``>=3.0.0,<4``
+``watchtower``                           ``>=2.0.1,<4``
 ``jsonpath_ng``                          ``>=1.5.3``
 ``redshift_connector``                   ``>=2.0.918``
 ``sqlalchemy_redshift``                  ``>=0.8.6``
-``asgiref``                              ``>=2.3.0``
+``asgiref``
 ``PyAthena``                             ``>=3.0.10``
-``jmespath``                             ``>=0.7.0``
+``jmespath``
 =======================================  ==================
 
 Cross provider package dependencies

{apache_airflow_providers_amazon-8.24.0 → apache_airflow_providers_amazon-8.24.0rc1}/README.rst

@@ -42,7 +42,7 @@
 
 Package ``apache-airflow-providers-amazon``
 
-Release: ``8.24.0``
+Release: ``8.24.0.rc1``
 
 
 Amazon integration (including `Amazon Web Services (AWS) <https://aws.amazon.com/>`__).
@@ -75,16 +75,16 @@ PIP package                              Version required
 ``apache-airflow``                       ``>=2.7.0``
 ``apache-airflow-providers-common-sql``  ``>=1.3.1``
 ``apache-airflow-providers-http``
-``boto3``                                ``>=1.34.90``
-``botocore``                             ``>=1.34.90``
+``boto3``                                ``>=1.33.0``
+``botocore``                             ``>=1.33.0``
 ``inflection``                           ``>=0.5.1``
-``watchtower``                           ``>=3.0.0,<4``
+``watchtower``                           ``>=2.0.1,<4``
 ``jsonpath_ng``                          ``>=1.5.3``
 ``redshift_connector``                   ``>=2.0.918``
 ``sqlalchemy_redshift``                  ``>=0.8.6``
-``asgiref``                              ``>=2.3.0``
+``asgiref``
 ``PyAthena``                             ``>=3.0.10``
-``jmespath``                             ``>=0.7.0``
+``jmespath``
 =======================================  ==================
 
 Cross provider package dependencies

{apache_airflow_providers_amazon-8.24.0 → apache_airflow_providers_amazon-8.24.0rc1}/airflow/providers/amazon/LICENSE

@@ -215,7 +215,7 @@ Third party Apache 2.0 licenses
 
 The following components are provided under the Apache 2.0 License.
 See project link for details. The text of each license is also included
-at 3rd-party-licenses/LICENSE-[project].txt.
+at licenses/LICENSE-[project].txt.
 
     (ALv2 License) hue v4.3.0 (https://github.com/cloudera/hue/)
     (ALv2 License) jqclock v2.3.0 (https://github.com/JohnRDOrazio/jQuery-Clock-Plugin)
@@ -227,7 +227,7 @@ MIT licenses
 ========================================================================
 
 The following components are provided under the MIT License. See project link for details.
-The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
+The text of each license is also included at licenses/LICENSE-[project].txt.
 
     (MIT License) jquery v3.5.1 (https://jquery.org/license/)
     (MIT License) dagre-d3 v0.6.4 (https://github.com/cpettitt/dagre-d3)
@@ -243,11 +243,11 @@ The text of each license is also included at 3rd-party-licenses/LICENSE-[project
 BSD 3-Clause licenses
 ========================================================================
 The following components are provided under the BSD 3-Clause license. See project links for details.
-The text of each license is also included at 3rd-party-licenses/LICENSE-[project].txt.
+The text of each license is also included at licenses/LICENSE-[project].txt.
 
     (BSD 3 License) d3 v5.16.0 (https://d3js.org)
     (BSD 3 License) d3-shape v2.1.0 (https://github.com/d3/d3-shape)
     (BSD 3 License) cgroupspy 0.2.1 (https://github.com/cloudsigma/cgroupspy)
 
 ========================================================================
-See 3rd-party-licenses/LICENSES-ui.txt for packages used in `/airflow/www`
+See licenses/LICENSES-ui.txt for packages used in `/airflow/www`

{apache_airflow_providers_amazon-8.24.0 → apache_airflow_providers_amazon-8.24.0rc1}/airflow/providers/amazon/aws/hooks/base_aws.py

@@ -156,9 +156,7 @@ class BaseSessionFactory(LoggingMixin):
 
         return async_get_session()
 
-    def create_session(
-        self, deferrable: bool = False
-    ) -> boto3.session.Session | aiobotocore.session.AioSession:
+    def create_session(self, deferrable: bool = False) -> boto3.session.Session:
         """Create boto3 or aiobotocore Session from connection config."""
         if not self.conn:
             self.log.info(
@@ -200,7 +198,7 @@ class BaseSessionFactory(LoggingMixin):
 
     def _create_session_with_assume_role(
         self, session_kwargs: dict[str, Any], deferrable: bool = False
-    ) -> boto3.session.Session | aiobotocore.session.AioSession:
+    ) -> boto3.session.Session:
         if self.conn.assume_role_method == "assume_role_with_web_identity":
             # Deferred credentials have no initial credentials
             credential_fetcher = self._get_web_identity_credential_fetcher()
@@ -241,10 +239,7 @@ class BaseSessionFactory(LoggingMixin):
         session._credentials = credentials
         session.set_config_variable("region", self.basic_session.region_name)
 
-        if not deferrable:
-            return boto3.session.Session(botocore_session=session, **session_kwargs)
-
-        return session
+        return boto3.session.Session(botocore_session=session, **session_kwargs)
 
     def _refresh_credentials(self) -> dict[str, Any]:
         self.log.debug("Refreshing credentials")

{apache_airflow_providers_amazon-8.24.0 → apache_airflow_providers_amazon-8.24.0rc1}/airflow/providers/amazon/aws/hooks/glue.py

@@ -20,7 +20,6 @@ from __future__ import annotations
 import asyncio
 import time
 from functools import cached_property
-from typing import Any
 
 from botocore.exceptions import ClientError
 
@@ -431,125 +430,3 @@ class GlueJobHook(AwsBaseHook):
             self.conn.create_job(**config)
 
         return self.job_name
-
-
-class GlueDataQualityHook(AwsBaseHook):
-    """
-    Interact with AWS Glue Data Quality.
-
-    Provide thick wrapper around :external+boto3:py:class:`boto3.client("glue") <Glue.Client>`.
-
-    Additional arguments (such as ``aws_conn_id``) may be specified and
-    are passed down to the underlying AwsBaseHook.
-
-    .. seealso::
-        - :class:`airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook`
-    """
-
-    def __init__(
-        self,
-        *args,
-        **kwargs,
-    ):
-        kwargs["client_type"] = "glue"
-        super().__init__(*args, **kwargs)
-
-    def has_data_quality_ruleset(self, name: str) -> bool:
-        try:
-            self.conn.get_data_quality_ruleset(Name=name)
-            return True
-        except self.conn.exceptions.EntityNotFoundException:
-            return False
-
-    def _log_results(self, result: dict[str, Any]) -> None:
-        """
-        Print the outcome of evaluation run, An evaluation run can involve multiple rulesets evaluated against a data source (Glue table).
-
-        Name    Description                                     Result        EvaluatedMetrics                                                                    EvaluationMessage
-        Rule_1    RowCount between 150000 and 600000             PASS        {'Dataset.*.RowCount': 300000.0}                                                       NaN
-        Rule_2    IsComplete "marketplace"                       PASS        {'Column.marketplace.Completeness': 1.0}                                               NaN
-        Rule_3    ColumnLength "marketplace" between 1 and 2     FAIL        {'Column.marketplace.MaximumLength': 9.0, 'Column.marketplace.MinimumLength': 3.0}     Value: 9.0 does not meet the constraint requirement!
-
-        """
-        import pandas as pd
-
-        pd.set_option("display.max_rows", None)
-        pd.set_option("display.max_columns", None)
-        pd.set_option("display.width", None)
-        pd.set_option("display.max_colwidth", None)
-
-        self.log.info(
-            "AWS Glue data quality ruleset evaluation result for RulesetName: %s RulesetEvaluationRunId: %s Score: %s",
-            result.get("RulesetName"),
-            result.get("RulesetEvaluationRunId"),
-            result.get("Score"),
-        )
-
-        rule_results = result["RuleResults"]
-        rule_results_df = pd.DataFrame(rule_results)
-        self.log.info(rule_results_df)
-
-    def get_evaluation_run_results(self, run_id: str) -> dict[str, Any]:
-        response = self.conn.get_data_quality_ruleset_evaluation_run(RunId=run_id)
-
-        return self.conn.batch_get_data_quality_result(ResultIds=response["ResultIds"])
-
-    def validate_evaluation_run_results(
-        self, evaluation_run_id: str, show_results: bool = True, verify_result_status: bool = True
-    ) -> None:
-        results = self.get_evaluation_run_results(evaluation_run_id)
-        total_failed_rules = 0
-
-        if results.get("ResultsNotFound"):
-            self.log.info(
-                "AWS Glue data quality ruleset evaluation run, results not found for %s",
-                results["ResultsNotFound"],
-            )
-
-        for result in results["Results"]:
-            rule_results = result["RuleResults"]
-
-            total_failed_rules += len(
-                list(
-                    filter(
-                        lambda result: result.get("Result") == "FAIL" or result.get("Result") == "ERROR",
-                        rule_results,
-                    )
-                )
-            )
-
-            if show_results:
-                self._log_results(result)
-
-        self.log.info(
-            "AWS Glue data quality ruleset evaluation run, total number of rules failed: %s",
-            total_failed_rules,
-        )
-
-        if verify_result_status and total_failed_rules > 0:
-            raise AirflowException(
-                "AWS Glue data quality ruleset evaluation run failed for one or more rules"
-            )
-
-    def log_recommendation_results(self, run_id: str) -> None:
-        """
-        Print the outcome of recommendation run, recommendation run generates multiple rules against a data source (Glue table) in Data Quality Definition Language (DQDL) format.
-
-        Rules = [
-        IsComplete "NAME",
-        ColumnLength "EMP_ID" between 1 and 12,
-        IsUnique "EMP_ID",
-        ColumnValues "INCOME" > 50000
-        ]
-        """
-        result = self.conn.get_data_quality_rule_recommendation_run(RunId=run_id)
-
-        if result.get("RecommendedRuleset"):
-            self.log.info(
-                "AWS Glue data quality recommended rules for DatabaseName: %s TableName: %s",
-                result["DataSource"]["GlueTable"]["DatabaseName"],
-                result["DataSource"]["GlueTable"]["TableName"],
-            )
-            self.log.info(result["RecommendedRuleset"])
-        else:
-            self.log.info("AWS Glue data quality, no recommended rules available for RunId: %s", run_id)

{apache_airflow_providers_amazon-8.24.0 → apache_airflow_providers_amazon-8.24.0rc1}/airflow/providers/amazon/aws/operators/bedrock.py

@@ -20,6 +20,7 @@ import json
 from time import sleep
 from typing import TYPE_CHECKING, Any, Sequence
 
+import botocore
 from botocore.exceptions import ClientError
 
 from airflow.configuration import conf
@@ -37,7 +38,7 @@ from airflow.providers.amazon.aws.triggers.bedrock import (
     BedrockKnowledgeBaseActiveTrigger,
     BedrockProvisionModelThroughputCompletedTrigger,
 )
-from airflow.providers.amazon.aws.utils import validate_execute_complete_event
+from airflow.providers.amazon.aws.utils import get_botocore_version, validate_execute_complete_event
 from airflow.providers.amazon.aws.utils.mixins import aws_template_fields
 from airflow.utils.helpers import prune_dict
 from airflow.utils.timezone import utcnow
@@ -798,11 +799,24 @@ class BedrockRaGOperator(AwsBaseOperator[BedrockAgentRuntimeHook]):
     def execute(self, context: Context) -> Any:
         self.validate_inputs()
 
-        result = self.hook.conn.retrieve_and_generate(
-            input={"text": self.input},
-            retrieveAndGenerateConfiguration=self.build_rag_config(),
-            **self.rag_kwargs,
-        )
+        try:
+            result = self.hook.conn.retrieve_and_generate(
+                input={"text": self.input},
+                retrieveAndGenerateConfiguration=self.build_rag_config(),
+                **self.rag_kwargs,
+            )
+        except botocore.exceptions.ParamValidationError as error:
+            if (
+                'Unknown parameter in retrieveAndGenerateConfiguration: "externalSourcesConfiguration"'
+                in str(error)
+            ) and (self.source_type == "EXTERNAL_SOURCES"):
+                self.log.error(
+                    "You are attempting to use External Sources and the BOTO API returned an "
+                    "error message which may indicate the need to update botocore to do this.  \n"
+                    "Support for External Sources was added in botocore 1.34.90 and you are using botocore %s",
+                    ".".join(map(str, get_botocore_version())),
+                )
+            raise
 
         self.log.info(
             "\nPrompt: %s\nResponse: %s\nCitations: %s",

{apache_airflow_providers_amazon-8.24.0 → apache_airflow_providers_amazon-8.24.0rc1}/airflow/providers/amazon/aws/operators/emr.py

@@ -263,34 +263,30 @@ class EmrStartNotebookExecutionOperator(BaseOperator):
         wait_for_completion: bool = False,
         aws_conn_id: str | None = "aws_default",
         # TODO: waiter_max_attempts and waiter_delay should default to None when the other two are deprecated.
-        waiter_max_attempts: int | None = None,
-        waiter_delay: int | None = None,
-        waiter_countdown: int | None = None,
-        waiter_check_interval_seconds: int | None = None,
+        waiter_max_attempts: int | None | ArgNotSet = NOTSET,
+        waiter_delay: int | None | ArgNotSet = NOTSET,
+        waiter_countdown: int = 25 * 60,
+        waiter_check_interval_seconds: int = 60,
         **kwargs: Any,
     ):
-        if waiter_check_interval_seconds:
+        if waiter_max_attempts is NOTSET:
             warnings.warn(
-                "The parameter `waiter_check_interval_seconds` has been deprecated to "
-                "standardize naming conventions.  Please `use waiter_delay instead`.  In the "
+                "The parameter waiter_countdown has been deprecated to standardize "
+                "naming conventions.  Please use waiter_max_attempts instead.  In the "
                 "future this will default to None and defer to the waiter's default value.",
                 AirflowProviderDeprecationWarning,
                 stacklevel=2,
             )
-        else:
-            waiter_check_interval_seconds = 60
-        if waiter_countdown:
+            waiter_max_attempts = waiter_countdown // waiter_check_interval_seconds
+        if waiter_delay is NOTSET:
             warnings.warn(
-                "The parameter waiter_countdown has been deprecated to standardize "
-                "naming conventions.  Please use waiter_max_attempts instead.  In the "
+                "The parameter waiter_check_interval_seconds has been deprecated to "
+                "standardize naming conventions.  Please use waiter_delay instead.  In the "
                 "future this will default to None and defer to the waiter's default value.",
                 AirflowProviderDeprecationWarning,
                 stacklevel=2,
             )
-            # waiter_countdown defaults to never timing out, which is not supported
-            # by boto waiters, so we will set it here to "a very long time" for now.
-            waiter_max_attempts = (waiter_countdown or 999) // waiter_check_interval_seconds
-
+            waiter_delay = waiter_check_interval_seconds
         super().__init__(**kwargs)
         self.editor_id = editor_id
         self.relative_path = relative_path
@@ -302,8 +298,8 @@ class EmrStartNotebookExecutionOperator(BaseOperator):
         self.wait_for_completion = wait_for_completion
         self.cluster_id = cluster_id
         self.aws_conn_id = aws_conn_id
-        self.waiter_max_attempts = waiter_max_attempts or 25
-        self.waiter_delay = waiter_delay or waiter_check_interval_seconds or 60
+        self.waiter_max_attempts = waiter_max_attempts
+        self.waiter_delay = waiter_delay
         self.master_instance_security_group_id = master_instance_security_group_id
 
     def execute(self, context: Context):
@@ -391,40 +387,36 @@ class EmrStopNotebookExecutionOperator(BaseOperator):
         wait_for_completion: bool = False,
         aws_conn_id: str | None = "aws_default",
         # TODO: waiter_max_attempts and waiter_delay should default to None when the other two are deprecated.
-        waiter_max_attempts: int | None = None,
-        waiter_delay: int | None = None,
-        waiter_countdown: int | None = None,
-        waiter_check_interval_seconds: int | None = None,
+        waiter_max_attempts: int | None | ArgNotSet = NOTSET,
+        waiter_delay: int | None | ArgNotSet = NOTSET,
+        waiter_countdown: int = 25 * 60,
+        waiter_check_interval_seconds: int = 60,
         **kwargs: Any,
     ):
-        if waiter_check_interval_seconds:
+        if waiter_max_attempts is NOTSET:
             warnings.warn(
-                "The parameter `waiter_check_interval_seconds` has been deprecated to "
-                "standardize naming conventions.  Please `use waiter_delay instead`.  In the "
+                "The parameter waiter_countdown has been deprecated to standardize "
+                "naming conventions.  Please use waiter_max_attempts instead.  In the "
                 "future this will default to None and defer to the waiter's default value.",
                 AirflowProviderDeprecationWarning,
                 stacklevel=2,
             )
-        else:
-            waiter_check_interval_seconds = 60
-        if waiter_countdown:
+            waiter_max_attempts = waiter_countdown // waiter_check_interval_seconds
+        if waiter_delay is NOTSET:
             warnings.warn(
-                "The parameter waiter_countdown has been deprecated to standardize "
-                "naming conventions.  Please use waiter_max_attempts instead.  In the "
+                "The parameter waiter_check_interval_seconds has been deprecated to "
+                "standardize naming conventions.  Please use waiter_delay instead.  In the "
                 "future this will default to None and defer to the waiter's default value.",
                 AirflowProviderDeprecationWarning,
                 stacklevel=2,
             )
-            # waiter_countdown defaults to never timing out, which is not supported
-            # by boto waiters, so we will set it here to "a very long time" for now.
-            waiter_max_attempts = (waiter_countdown or 999) // waiter_check_interval_seconds
-
+            waiter_delay = waiter_check_interval_seconds
         super().__init__(**kwargs)
         self.notebook_execution_id = notebook_execution_id
         self.wait_for_completion = wait_for_completion
         self.aws_conn_id = aws_conn_id
-        self.waiter_max_attempts = waiter_max_attempts or 25
-        self.waiter_delay = waiter_delay or waiter_check_interval_seconds or 60
+        self.waiter_max_attempts = waiter_max_attempts
+        self.waiter_delay = waiter_delay
 
     def execute(self, context: Context) -> None:
         emr_hook = EmrHook(aws_conn_id=self.aws_conn_id)
@@ -830,8 +822,8 @@ class EmrCreateJobFlowOperator(BaseOperator):
                 trigger=EmrCreateJobFlowTrigger(
                     job_flow_id=self._job_flow_id,
                     aws_conn_id=self.aws_conn_id,
-                    waiter_delay=self.waiter_delay,
-                    waiter_max_attempts=self.waiter_max_attempts,
+                    poll_interval=self.waiter_delay,
+                    max_attempts=self.waiter_max_attempts,
                 ),
                 method_name="execute_complete",
                 # timeout is set to ensure that if a trigger dies, the timeout does not restart