apache-airflow-providers-google 10.21.1rc2__tar.gz → 10.22.0__tar.gz

{apache_airflow_providers_google-10.21.1rc2 → apache_airflow_providers_google-10.22.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: apache-airflow-providers-google
-Version: 10.21.1rc2
+Version: 10.22.0
 Summary: Provider package apache-airflow-providers-google for Apache Airflow
 Keywords: airflow-provider,google,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -22,12 +22,12 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: System :: Monitoring
 Requires-Dist: PyOpenSSL>=23.0.0
-Requires-Dist: apache-airflow-providers-common-compat>=1.1.0rc0
-Requires-Dist: apache-airflow-providers-common-sql>=1.7.2rc0
-Requires-Dist: apache-airflow>=2.7.0rc0
+Requires-Dist: apache-airflow-providers-common-compat>=1.1.0
+Requires-Dist: apache-airflow-providers-common-sql>=1.7.2
+Requires-Dist: apache-airflow>=2.8.0
 Requires-Dist: asgiref>=3.5.2
 Requires-Dist: dill>=0.2.3
-Requires-Dist: gcloud-aio-auth>=4.0.0,<5.0.0
+Requires-Dist: gcloud-aio-auth>=5.2.0
 Requires-Dist: gcloud-aio-bigquery>=6.1.2
 Requires-Dist: gcloud-aio-storage>=9.0.0
 Requires-Dist: gcsfs>=2023.10.0
@@ -86,20 +86,20 @@ Requires-Dist: python-slugify>=7.0.0
 Requires-Dist: sqlalchemy-bigquery>=1.2.1
 Requires-Dist: sqlalchemy-spanner>=1.6.2
 Requires-Dist: tenacity>=8.1.0
-Requires-Dist: apache-airflow-providers-amazon>=2.6.0rc0 ; extra == "amazon"
+Requires-Dist: apache-airflow-providers-amazon>=2.6.0 ; extra == "amazon"
 Requires-Dist: apache-airflow-providers-apache-beam ; extra == "apache.beam"
 Requires-Dist: apache-beam[gcp] ; extra == "apache.beam"
 Requires-Dist: apache-airflow-providers-apache-cassandra ; extra == "apache.cassandra"
-Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0rc0 ; extra == "cncf.kubernetes"
+Requires-Dist: apache-airflow-providers-cncf-kubernetes>=7.2.0 ; extra == "cncf.kubernetes"
 Requires-Dist: apache-airflow-providers-common-compat ; extra == "common.compat"
 Requires-Dist: apache-airflow-providers-common-sql ; extra == "common.sql"
-Requires-Dist: apache-airflow-providers-facebook>=2.2.0rc0 ; extra == "facebook"
+Requires-Dist: apache-airflow-providers-facebook>=2.2.0 ; extra == "facebook"
 Requires-Dist: plyvel ; extra == "leveldb"
 Requires-Dist: apache-airflow-providers-microsoft-azure ; extra == "microsoft.azure"
 Requires-Dist: apache-airflow-providers-microsoft-mssql ; extra == "microsoft.mssql"
 Requires-Dist: apache-airflow-providers-mysql ; extra == "mysql"
 Requires-Dist: apache-airflow-providers-openlineage ; extra == "openlineage"
-Requires-Dist: apache-airflow-providers-oracle>=3.1.0rc0 ; extra == "oracle"
+Requires-Dist: apache-airflow-providers-oracle>=3.1.0 ; extra == "oracle"
 Requires-Dist: apache-airflow-providers-postgres ; extra == "postgres"
 Requires-Dist: apache-airflow-providers-presto ; extra == "presto"
 Requires-Dist: apache-airflow-providers-salesforce ; extra == "salesforce"
@@ -107,8 +107,8 @@ Requires-Dist: apache-airflow-providers-sftp ; extra == "sftp"
 Requires-Dist: apache-airflow-providers-ssh ; extra == "ssh"
 Requires-Dist: apache-airflow-providers-trino ; extra == "trino"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-google/10.21.1/changelog.html
-Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-google/10.21.1
+Project-URL: Changelog, https://airflow.apache.org/docs/apache-airflow-providers-google/10.22.0/changelog.html
+Project-URL: Documentation, https://airflow.apache.org/docs/apache-airflow-providers-google/10.22.0
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
 Project-URL: Twitter, https://twitter.com/ApacheAirflow
@@ -177,7 +177,7 @@ Provides-Extra: trino
 
 Package ``apache-airflow-providers-google``
 
-Release: ``10.21.1.rc2``
+Release: ``10.22.0``
 
 
 Google services including:
@@ -197,7 +197,7 @@ This is a provider package for ``google`` provider. All classes for this provide
 are in ``airflow.providers.google`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/10.21.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/10.22.0/>`_.
 
 Installation
 ------------
@@ -214,12 +214,12 @@ Requirements
 ==========================================  =========================================
 PIP package                                 Version required
 ==========================================  =========================================
-``apache-airflow``                          ``>=2.7.0``
+``apache-airflow``                          ``>=2.8.0``
 ``apache-airflow-providers-common-compat``  ``>=1.1.0``
 ``apache-airflow-providers-common-sql``     ``>=1.7.2``
 ``asgiref``                                 ``>=3.5.2``
 ``dill``                                    ``>=0.2.3``
-``gcloud-aio-auth``                         ``>=4.0.0,<5.0.0``
+``gcloud-aio-auth``                         ``>=5.2.0``
 ``gcloud-aio-bigquery``                     ``>=6.1.2``
 ``gcloud-aio-storage``                      ``>=9.0.0``
 ``gcsfs``                                   ``>=2023.10.0``
@@ -318,4 +318,4 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/10.21.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/10.22.0/changelog.html>`_.

{apache_airflow_providers_google-10.21.1rc2 → apache_airflow_providers_google-10.22.0}/README.rst

@@ -42,7 +42,7 @@
 
 Package ``apache-airflow-providers-google``
 
-Release: ``10.21.1.rc2``
+Release: ``10.22.0``
 
 
 Google services including:
@@ -62,7 +62,7 @@ This is a provider package for ``google`` provider. All classes for this provide
 are in ``airflow.providers.google`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/10.21.1/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/10.22.0/>`_.
 
 Installation
 ------------
@@ -79,12 +79,12 @@ Requirements
 ==========================================  =========================================
 PIP package                                 Version required
 ==========================================  =========================================
-``apache-airflow``                          ``>=2.7.0``
+``apache-airflow``                          ``>=2.8.0``
 ``apache-airflow-providers-common-compat``  ``>=1.1.0``
 ``apache-airflow-providers-common-sql``     ``>=1.7.2``
 ``asgiref``                                 ``>=3.5.2``
 ``dill``                                    ``>=0.2.3``
-``gcloud-aio-auth``                         ``>=4.0.0,<5.0.0``
+``gcloud-aio-auth``                         ``>=5.2.0``
 ``gcloud-aio-bigquery``                     ``>=6.1.2``
 ``gcloud-aio-storage``                      ``>=9.0.0``
 ``gcsfs``                                   ``>=2023.10.0``
@@ -183,4 +183,4 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/10.21.1/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/10.22.0/changelog.html>`_.

{apache_airflow_providers_google-10.21.1rc2 → apache_airflow_providers_google-10.22.0}/airflow/providers/google/__init__.py

@@ -29,11 +29,11 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "10.21.1"
+__version__ = "10.22.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
-    "2.7.0"
+    "2.8.0"
 ):
     raise RuntimeError(
-        f"The package `apache-airflow-providers-google:{__version__}` needs Apache Airflow 2.7.0+"
+        f"The package `apache-airflow-providers-google:{__version__}` needs Apache Airflow 2.8.0+"
     )

{apache_airflow_providers_google-10.21.1rc2 → apache_airflow_providers_google-10.22.0}/airflow/providers/google/cloud/hooks/cloud_run.py

@@ -22,13 +22,19 @@ from typing import TYPE_CHECKING, Any, Iterable, Sequence
 
 from google.cloud.run_v2 import (
     CreateJobRequest,
+    CreateServiceRequest,
     DeleteJobRequest,
+    DeleteServiceRequest,
     GetJobRequest,
+    GetServiceRequest,
     Job,
     JobsAsyncClient,
     JobsClient,
     ListJobsRequest,
     RunJobRequest,
+    Service,
+    ServicesAsyncClient,
+    ServicesClient,
     UpdateJobRequest,
 )
 from google.longrunning import operations_pb2  # type: ignore[attr-defined]
@@ -39,6 +45,7 @@ from airflow.providers.google.common.hooks.base_google import PROVIDE_PROJECT_ID
 
 if TYPE_CHECKING:
     from google.api_core import operation
+    from google.api_core.operation_async import AsyncOperation
     from google.cloud.run_v2.services.jobs import pagers
 
 
@@ -183,3 +190,120 @@ class CloudRunAsyncHook(GoogleBaseHook):
         return await self.get_conn().get_operation(
             operations_pb2.GetOperationRequest(name=operation_name), timeout=120
         )
+
+
+class CloudRunServiceHook(GoogleBaseHook):
+    """
+    Hook for the Google Cloud Run services.
+
+    :param gcp_conn_id: The connection ID to use when fetching connection info.
+    :param impersonation_chain: Optional service account to impersonate using short-term
+        credentials, or chained list of accounts required to get the access_token
+        of the last account in the list, which will be impersonated in the request.
+        If set as a string, the account must grant the originating account
+        the Service Account Token Creator IAM role.
+        If set as a sequence, the identities from the list must grant
+        Service Account Token Creator IAM role to the directly preceding identity, with first
+        account from the list granting this role to the originating account.
+    """
+
+    def __init__(
+        self,
+        gcp_conn_id: str = "google_cloud_default",
+        impersonation_chain: str | Sequence[str] | None = None,
+    ):
+        self._client: ServicesClient | None = None
+        super().__init__(gcp_conn_id=gcp_conn_id, impersonation_chain=impersonation_chain)
+
+    def get_conn(self):
+        if self._client is None:
+            self._client = ServicesClient(credentials=self.get_credentials(), client_info=CLIENT_INFO)
+
+        return self._client
+
+    @GoogleBaseHook.fallback_to_default_project_id
+    def get_service(self, service_name: str, region: str, project_id: str = PROVIDE_PROJECT_ID):
+        get_service_request = GetServiceRequest(
+            name=f"projects/{project_id}/locations/{region}/services/{service_name}"
+        )
+        return self.get_conn().get_service(get_service_request)
+
+    @GoogleBaseHook.fallback_to_default_project_id
+    def create_service(
+        self, service_name: str, service: Service | dict, region: str, project_id: str = PROVIDE_PROJECT_ID
+    ) -> Service:
+        if isinstance(service, dict):
+            service = Service(service)
+
+        create_request = CreateServiceRequest(
+            parent=f"projects/{project_id}/locations/{region}",
+            service=service,
+            service_id=service_name,
+        )
+
+        operation = self.get_conn().create_service(create_request)
+        return operation.result()
+
+    @GoogleBaseHook.fallback_to_default_project_id
+    def delete_service(self, service_name: str, region: str, project_id: str = PROVIDE_PROJECT_ID) -> Service:
+        delete_request = DeleteServiceRequest(
+            name=f"projects/{project_id}/locations/{region}/services/{service_name}"
+        )
+
+        operation = self.get_conn().delete_service(delete_request)
+        return operation.result()
+
+
+class CloudRunServiceAsyncHook(GoogleBaseHook):
+    """
+    Async hook for the Google Cloud Run services.
+
+    :param gcp_conn_id: The connection ID to use when fetching connection info.
+    :param impersonation_chain: Optional service account to impersonate using short-term
+        credentials, or chained list of accounts required to get the access_token
+        of the last account in the list, which will be impersonated in the request.
+        If set as a string, the account must grant the originating account
+        the Service Account Token Creator IAM role.
+        If set as a sequence, the identities from the list must grant
+        Service Account Token Creator IAM role to the directly preceding identity, with first
+        account from the list granting this role to the originating account.
+    """
+
+    def __init__(
+        self,
+        gcp_conn_id: str = "google_cloud_default",
+        impersonation_chain: str | Sequence[str] | None = None,
+    ):
+        self._client: ServicesClient | None = None
+        super().__init__(gcp_conn_id=gcp_conn_id, impersonation_chain=impersonation_chain)
+
+    def get_conn(self):
+        if self._client is None:
+            self._client = ServicesAsyncClient(credentials=self.get_credentials(), client_info=CLIENT_INFO)
+
+        return self._client
+
+    @GoogleBaseHook.fallback_to_default_project_id
+    async def create_service(
+        self, service_name: str, service: Service | dict, region: str, project_id: str = PROVIDE_PROJECT_ID
+    ) -> AsyncOperation:
+        if isinstance(service, dict):
+            service = Service(service)
+
+        create_request = CreateServiceRequest(
+            parent=f"projects/{project_id}/locations/{region}",
+            service=service,
+            service_id=service_name,
+        )
+
+        return await self.get_conn().create_service(create_request)
+
+    @GoogleBaseHook.fallback_to_default_project_id
+    async def delete_service(
+        self, service_name: str, region: str, project_id: str = PROVIDE_PROJECT_ID
+    ) -> AsyncOperation:
+        delete_request = DeleteServiceRequest(
+            name=f"projects/{project_id}/locations/{region}/services/{service_name}"
+        )
+
+        return await self.get_conn().delete_service(delete_request)

{apache_airflow_providers_google-10.21.1rc2 → apache_airflow_providers_google-10.22.0}/airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py

@@ -552,6 +552,8 @@ class CloudDataTransferServiceHook(GoogleBaseHook):
 class CloudDataTransferServiceAsyncHook(GoogleBaseAsyncHook):
     """Asynchronous hook for Google Storage Transfer Service."""
 
+    sync_hook_class = CloudDataTransferServiceHook
+
     def __init__(self, project_id: str = PROVIDE_PROJECT_ID, **kwargs: Any) -> None:
         super().__init__(**kwargs)
         self.project_id = project_id

{apache_airflow_providers_google-10.21.1rc2 → apache_airflow_providers_google-10.22.0}/airflow/providers/google/cloud/operators/cloud_run.py

@@ -19,11 +19,13 @@ from __future__ import annotations
 
 from typing import TYPE_CHECKING, Any, Sequence
 
-from google.cloud.run_v2 import Job
+import google.cloud.exceptions
+from google.api_core.exceptions import AlreadyExists
+from google.cloud.run_v2 import Job, Service
 
 from airflow.configuration import conf
 from airflow.exceptions import AirflowException
-from airflow.providers.google.cloud.hooks.cloud_run import CloudRunHook
+from airflow.providers.google.cloud.hooks.cloud_run import CloudRunHook, CloudRunServiceHook
 from airflow.providers.google.cloud.operators.cloud_base import GoogleCloudBaseOperator
 from airflow.providers.google.cloud.triggers.cloud_run import CloudRunJobFinishedTrigger, RunJobStatus
 
@@ -353,3 +355,142 @@ class CloudRunExecuteJobOperator(GoogleCloudBaseOperator):
         except Exception:
             error = operation.exception(timeout=self.timeout_seconds)
             raise AirflowException(error)
+
+
+class CloudRunCreateServiceOperator(GoogleCloudBaseOperator):
+    """
+    Creates a Service without executing it. Pushes the created service to xcom.
+
+    :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+    :param region: Required. The ID of the Google Cloud region that the service belongs to.
+    :param service_name: Required. The name of the service to create.
+    :param service: The service descriptor containing the configuration of the service to submit.
+    :param gcp_conn_id: The connection ID used to connect to Google Cloud.
+    :param impersonation_chain: Optional service account to impersonate using short-term
+        credentials, or chained list of accounts required to get the access_token
+        of the last account in the list, which will be impersonated in the request.
+        If set as a string, the account must grant the originating account
+        the Service Account Token Creator IAM role.
+        If set as a sequence, the identities from the list must grant
+        Service Account Token Creator IAM role to the directly preceding identity, with first
+        account from the list granting this role to the originating account (templated).
+    """
+
+    template_fields = ("project_id", "region", "gcp_conn_id", "impersonation_chain", "service_name")
+
+    def __init__(
+        self,
+        project_id: str,
+        region: str,
+        service_name: str,
+        service: dict | Service,
+        gcp_conn_id: str = "google_cloud_default",
+        impersonation_chain: str | Sequence[str] | None = None,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.project_id = project_id
+        self.region = region
+        self.service = service
+        self.service_name = service_name
+        self.gcp_conn_id = gcp_conn_id
+        self.impersonation_chain = impersonation_chain
+        self._validate_inputs()
+
+    def _validate_inputs(self):
+        missing_fields = [k for k in ["project_id", "region", "service_name"] if not getattr(self, k)]
+        if not self.project_id or not self.region or not self.service_name:
+            raise AirflowException(
+                f"Required parameters are missing: {missing_fields}. These parameters be passed either as "
+                "keyword parameter or as extra field in Airflow connection definition. Both are not set!"
+            )
+
+    def execute(self, context: Context):
+        hook: CloudRunServiceHook = CloudRunServiceHook(
+            gcp_conn_id=self.gcp_conn_id, impersonation_chain=self.impersonation_chain
+        )
+
+        try:
+            service = hook.create_service(
+                service=self.service,
+                service_name=self.service_name,
+                region=self.region,
+                project_id=self.project_id,
+            )
+        except AlreadyExists:
+            self.log.info(
+                "Already existed Cloud run service, service_name=%s, region=%s",
+                self.service_name,
+                self.region,
+            )
+            return hook.get_service(
+                service_name=self.service_name, region=self.region, project_id=self.project_id
+            )
+        except google.cloud.exceptions.GoogleCloudError as e:
+            self.log.error("An error occurred. Exiting.")
+            raise e
+
+        return Service.to_dict(service)
+
+
+class CloudRunDeleteServiceOperator(GoogleCloudBaseOperator):
+    """
+    Deletes a Service without executing it. Pushes the deleted service to xcom.
+
+    :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+    :param region: Required. The ID of the Google Cloud region that the service belongs to.
+    :param service_name: Required. The name of the service to create.
+    :param gcp_conn_id: The connection ID used to connect to Google Cloud.
+    :param impersonation_chain: Optional service account to impersonate using short-term
+        credentials, or chained list of accounts required to get the access_token
+        of the last account in the list, which will be impersonated in the request.
+        If set as a string, the account must grant the originating account
+        the Service Account Token Creator IAM role.
+        If set as a sequence, the identities from the list must grant
+        Service Account Token Creator IAM role to the directly preceding identity, with first
+        account from the list granting this role to the originating account (templated).
+    """
+
+    template_fields = ("project_id", "region", "gcp_conn_id", "impersonation_chain", "service_name")
+
+    def __init__(
+        self,
+        project_id: str,
+        region: str,
+        service_name: str,
+        gcp_conn_id: str = "google_cloud_default",
+        impersonation_chain: str | Sequence[str] | None = None,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.project_id = project_id
+        self.region = region
+        self.service_name = service_name
+        self.gcp_conn_id = gcp_conn_id
+        self.impersonation_chain = impersonation_chain
+        self._validate_inputs()
+
+    def _validate_inputs(self):
+        missing_fields = [k for k in ["project_id", "region", "service_name"] if not getattr(self, k)]
+        if not self.project_id or not self.region or not self.service_name:
+            raise AirflowException(
+                f"Required parameters are missing: {missing_fields}. These parameters be passed either as "
+                "keyword parameter or as extra field in Airflow connection definition. Both are not set!"
+            )
+
+    def execute(self, context: Context):
+        hook: CloudRunServiceHook = CloudRunServiceHook(
+            gcp_conn_id=self.gcp_conn_id, impersonation_chain=self.impersonation_chain
+        )
+
+        try:
+            service = hook.delete_service(
+                service_name=self.service_name,
+                region=self.region,
+                project_id=self.project_id,
+            )
+        except google.cloud.exceptions.NotFound as e:
+            self.log.error("An error occurred. Not Found.")
+            raise e
+
+        return Service.to_dict(service)