apache-airflow-providers-google 17.0.0rc1__tar.gz → 17.1.0rc1__tar.gz

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: apache-airflow-providers-google
-Version: 17.0.0rc1
+Version: 17.1.0rc1
 Summary: Provider package apache-airflow-providers-google for Apache Airflow
 Keywords: airflow-provider,google,airflow,integration
 Author-email: Apache Software Foundation <dev@airflow.apache.org>
@@ -35,7 +35,7 @@ Requires-Dist: google-api-core>=2.11.0,!=2.16.0,!=2.18.0
 Requires-Dist: google-api-python-client>=2.0.2
 Requires-Dist: google-auth>=2.29.0
 Requires-Dist: google-auth-httplib2>=0.0.1
-Requires-Dist: google-cloud-aiplatform[evaluation]>=1.73.0
+Requires-Dist: google-cloud-aiplatform[evaluation]>=1.98.0
 Requires-Dist: ray[default]>=2.42.0 ; python_version < '3.13'
 Requires-Dist: google-cloud-bigquery-storage>=2.31.0 ; python_version < '3.13'
 Requires-Dist: google-cloud-alloydb>=0.4.0
@@ -113,8 +113,8 @@ Requires-Dist: apache-airflow-providers-sftp ; extra == "sftp"
 Requires-Dist: apache-airflow-providers-ssh ; extra == "ssh"
 Requires-Dist: apache-airflow-providers-trino ; extra == "trino"
 Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
-Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.0.0/changelog.html
-Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.0.0
+Project-URL: Changelog, https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.1.0/changelog.html
+Project-URL: Documentation, https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.1.0
 Project-URL: Mastodon, https://fosstodon.org/@airflow
 Project-URL: Slack Chat, https://s.apache.org/airflow-slack
 Project-URL: Source Code, https://github.com/apache/airflow
@@ -164,7 +164,7 @@ Provides-Extra: trino
 
 Package ``apache-airflow-providers-google``
 
-Release: ``17.0.0``
+Release: ``17.1.0``
 
 Release Date: ``|PypiReleaseDate|``
 
@@ -185,7 +185,7 @@ This is a provider package for ``google`` provider. All classes for this provide
 are in ``airflow.providers.google`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/17.0.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/17.1.0/>`_.
 
 Installation
 ------------
@@ -217,7 +217,7 @@ PIP package                                 Version required
 ``google-api-python-client``                ``>=2.0.2``
 ``google-auth``                             ``>=2.29.0``
 ``google-auth-httplib2``                    ``>=0.0.1``
-``google-cloud-aiplatform[evaluation]``     ``>=1.73.0``
+``google-cloud-aiplatform[evaluation]``     ``>=1.98.0``
 ``ray[default]``                            ``>=2.42.0; python_version < "3.13"``
 ``google-cloud-bigquery-storage``           ``>=2.31.0; python_version < "3.13"``
 ``google-cloud-alloydb``                    ``>=0.4.0``
@@ -314,5 +314,5 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/17.0.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/17.1.0/changelog.html>`_.
 

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/README.rst

@@ -23,7 +23,7 @@
 
 Package ``apache-airflow-providers-google``
 
-Release: ``17.0.0``
+Release: ``17.1.0``
 
 Release Date: ``|PypiReleaseDate|``
 
@@ -44,7 +44,7 @@ This is a provider package for ``google`` provider. All classes for this provide
 are in ``airflow.providers.google`` python package.
 
 You can find package information and changelog for the provider
-in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/17.0.0/>`_.
+in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-google/17.1.0/>`_.
 
 Installation
 ------------
@@ -76,7 +76,7 @@ PIP package                                 Version required
 ``google-api-python-client``                ``>=2.0.2``
 ``google-auth``                             ``>=2.29.0``
 ``google-auth-httplib2``                    ``>=0.0.1``
-``google-cloud-aiplatform[evaluation]``     ``>=1.73.0``
+``google-cloud-aiplatform[evaluation]``     ``>=1.98.0``
 ``ray[default]``                            ``>=2.42.0; python_version < "3.13"``
 ``google-cloud-bigquery-storage``           ``>=2.31.0; python_version < "3.13"``
 ``google-cloud-alloydb``                    ``>=0.4.0``
@@ -173,4 +173,4 @@ Dependent package
 ========================================================================================================================  ====================
 
 The changelog for the provider package can be found in the
-`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/17.0.0/changelog.html>`_.
+`changelog <https://airflow.apache.org/docs/apache-airflow-providers-google/17.1.0/changelog.html>`_.

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/docs/changelog.rst

@@ -27,6 +27,26 @@
 Changelog
 ---------
 
+17.1.0
+......
+
+Features
+~~~~~~~~
+
+* ``Add Private Service Connect interface support to VertexAI operators (#54170)``
+
+Bug Fixes
+~~~~~~~~~
+
+* ``Fix the google provider vertex_ai ImportDataOperator (#53865)``
+* ``Fix the google cloud provider RayHook protobuf compatibility (#54014)``
+* ``Fix the PostgresHook ignoring custom adapters registered (#53707)``
+
+.. Below changes are excluded from the changelog. Move them to
+   appropriate section above if needed. Do not delete the lines(!):
+   * ``Fix location and timeout for Workflows and Dataproc Metastore system tests (#54167)``
+   * ``Fix ray to be imported conditionally (tests is skipped on Python 3.13) (#54031)``
+
 17.0.0
 ......
 

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/docs/index.rst

@@ -88,7 +88,7 @@ Google services including:
   - `Google Workspace <https://workspace.google.com/>`__ (formerly Google Suite)
 
 
-Release: 17.0.0
+Release: 17.1.0
 
 Release Date: ``|PypiReleaseDate|``
 
@@ -128,7 +128,7 @@ PIP package                                 Version required
 ``google-api-python-client``                ``>=2.0.2``
 ``google-auth``                             ``>=2.29.0``
 ``google-auth-httplib2``                    ``>=0.0.1``
-``google-cloud-aiplatform[evaluation]``     ``>=1.73.0``
+``google-cloud-aiplatform[evaluation]``     ``>=1.98.0``
 ``ray[default]``                            ``>=2.42.0; python_version < "3.13"``
 ``google-cloud-bigquery-storage``           ``>=2.31.0; python_version < "3.13"``
 ``google-cloud-alloydb``                    ``>=0.4.0``
@@ -230,5 +230,5 @@ Downloading official packages
 You can download officially released packages and verify their checksums and signatures from the
 `Official Apache Download site <https://downloads.apache.org/airflow/providers/>`_
 
-* `The apache-airflow-providers-google 17.0.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.0.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.0.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.0.0.tar.gz.sha512>`__)
-* `The apache-airflow-providers-google 17.0.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.0.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.0.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.0.0-py3-none-any.whl.sha512>`__)
+* `The apache-airflow-providers-google 17.1.0 sdist package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.1.0.tar.gz>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.1.0.tar.gz.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.1.0.tar.gz.sha512>`__)
+* `The apache-airflow-providers-google 17.1.0 wheel package <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.1.0-py3-none-any.whl>`_ (`asc <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.1.0-py3-none-any.whl.asc>`__, `sha512 <https://downloads.apache.org/airflow/providers/apache_airflow_providers_google-17.1.0-py3-none-any.whl.sha512>`__)

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/docs/operators/cloud/vertex_ai.rst

@@ -861,6 +861,19 @@ To delete experiment run you can use
     :start-after: [START how_to_cloud_vertex_ai_delete_experiment_run_operator]
     :end-before: [END how_to_cloud_vertex_ai_delete_experiment_run_operator]
 
+Use Private Service Connect interface
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can configure Private Service Connect interface connections for
+:class:`~airflow.providers.google.cloud.operators.vertex_ai.custom_job.CreateCustomContainerTrainingJobOperator`,
+:class:`~airflow.providers.google.cloud.operators.vertex_ai.custom_job.CreateCustomPythonPackageTrainingJobOperator`,
+:class:`~airflow.providers.google.cloud.operators.vertex_ai.custom_job.CreateCustomTrainingJobOperator` and
+:class:`~airflow.providers.google.cloud.operators.vertex_ai.ray.CreateRayClusterOperator`
+operators in Vertex AI. For doing it you must first configure the PSC interface by following the provided
+`documentation <https://cloud.google.com/vertex-ai/docs/general/vpc-psc-i-setup>`__.
+Then, specify the PSC configuration in the ``psc_interface_config`` parameter.
+
+
 Reference
 ^^^^^^^^^
 

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/provider.yaml

@@ -29,12 +29,13 @@ description: |
     - `Google Workspace <https://workspace.google.com/>`__ (formerly Google Suite)
 
 state: ready
-source-date-epoch: 1753692233
+source-date-epoch: 1754503289
 # Note that those versions are maintained by release manager - do not update them manually
 # with the exception of case where other provider in sources has >= new provider version.
 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have
 # to be done in the same PR
 versions:
+  - 17.1.0
   - 17.0.0
   - 16.1.0
   - 16.0.0

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/pyproject.toml

@@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi"
 
 [project]
 name = "apache-airflow-providers-google"
-version = "17.0.0rc1"
+version = "17.1.0rc1"
 description = "Provider package apache-airflow-providers-google for Apache Airflow"
 readme = "README.rst"
 authors = [
@@ -78,7 +78,7 @@ dependencies = [
     # google-cloud-aiplatform doesn't install ray for python 3.12 (issue: https://github.com/googleapis/python-aiplatform/issues/5252).
     # Temporarily lock in ray 2.42.0 which is compatible with python 3.12 until linked issue is solved.
     # Remove the ray dependency as well as google-cloud-bigquery-storage once linked issue is fixed
-    "google-cloud-aiplatform[evaluation]>=1.73.0",
+    "google-cloud-aiplatform[evaluation]>=1.98.0",
     "ray[default]>=2.42.0 ; python_version < '3.13'",
     "google-cloud-bigquery-storage>=2.31.0 ; python_version < '3.13'",
     "google-cloud-alloydb>=0.4.0",
@@ -259,8 +259,8 @@ apache-airflow-providers-common-sql = {workspace = true}
 apache-airflow-providers-standard = {workspace = true}
 
 [project.urls]
-"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.0.0"
-"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.0.0/changelog.html"
+"Documentation" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.1.0"
+"Changelog" = "https://airflow.staged.apache.org/docs/apache-airflow-providers-google/17.1.0/changelog.html"
 "Bug Tracker" = "https://github.com/apache/airflow/issues"
 "Source Code" = "https://github.com/apache/airflow"
 "Slack Chat" = "https://s.apache.org/airflow-slack"

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/src/airflow/providers/google/__init__.py

@@ -29,7 +29,7 @@ from airflow import __version__ as airflow_version
 
 __all__ = ["__version__"]
 
-__version__ = "17.0.0"
+__version__ = "17.1.0"
 
 if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse(
     "2.10.0"

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/src/airflow/providers/google/cloud/hooks/vertex_ai/custom_job.py

@@ -55,7 +55,7 @@ if TYPE_CHECKING:
     from google.cloud.aiplatform_v1.services.pipeline_service.pagers import (
         ListTrainingPipelinesPager,
     )
-    from google.cloud.aiplatform_v1.types import CustomJob, TrainingPipeline
+    from google.cloud.aiplatform_v1.types import CustomJob, PscInterfaceConfig, TrainingPipeline
 
 
 class CustomJobHook(GoogleBaseHook, OperationHelper):
@@ -317,6 +317,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         is_default_version: bool | None = None,
         model_version_aliases: list[str] | None = None,
         model_version_description: str | None = None,
+        psc_interface_config: PscInterfaceConfig | None = None,
     ) -> tuple[models.Model | None, str, str]:
         """Run a training pipeline job and wait until its completion."""
         model = job.run(
@@ -350,6 +351,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
             is_default_version=is_default_version,
             model_version_aliases=model_version_aliases,
             model_version_description=model_version_description,
+            psc_interface_config=psc_interface_config,
         )
         training_id = self.extract_training_id(job.resource_name)
         custom_job_id = self.extract_custom_job_id(
@@ -574,6 +576,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         timestamp_split_column_name: str | None = None,
         tensorboard: str | None = None,
         sync=True,
+        psc_interface_config: PscInterfaceConfig | None = None,
     ) -> tuple[models.Model | None, str, str]:
         """
         Create Custom Container Training Job.
@@ -837,6 +840,8 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         :param sync: Whether to execute the AI Platform job synchronously. If False, this method
                 will be executed in concurrent Future and any downstream object will
                 be immediately returned and synced when the Future has completed.
+        :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+            training.
         """
         self._job = self.get_custom_container_training_job(
             project=project_id,
@@ -896,6 +901,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
             is_default_version=is_default_version,
             model_version_aliases=model_version_aliases,
             model_version_description=model_version_description,
+            psc_interface_config=psc_interface_config,
         )
 
         return model, training_id, custom_job_id
@@ -958,6 +964,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         model_version_aliases: list[str] | None = None,
         model_version_description: str | None = None,
         sync=True,
+        psc_interface_config: PscInterfaceConfig | None = None,
     ) -> tuple[models.Model | None, str, str]:
         """
         Create Custom Python Package Training Job.
@@ -1220,6 +1227,8 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         :param sync: Whether to execute the AI Platform job synchronously. If False, this method
                 will be executed in concurrent Future and any downstream object will
                 be immediately returned and synced when the Future has completed.
+        :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+            training.
         """
         self._job = self.get_custom_python_package_training_job(
             project=project_id,
@@ -1280,6 +1289,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
             is_default_version=is_default_version,
             model_version_aliases=model_version_aliases,
             model_version_description=model_version_description,
+            psc_interface_config=psc_interface_config,
         )
 
         return model, training_id, custom_job_id
@@ -1342,6 +1352,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         timestamp_split_column_name: str | None = None,
         tensorboard: str | None = None,
         sync=True,
+        psc_interface_config: PscInterfaceConfig | None = None,
     ) -> tuple[models.Model | None, str, str]:
         """
         Create Custom Training Job.
@@ -1604,6 +1615,8 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         :param sync: Whether to execute the AI Platform job synchronously. If False, this method
                 will be executed in concurrent Future and any downstream object will
                 be immediately returned and synced when the Future has completed.
+        :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+            training.
         """
         self._job = self.get_custom_training_job(
             project=project_id,
@@ -1664,6 +1677,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
             is_default_version=is_default_version,
             model_version_aliases=model_version_aliases,
             model_version_description=model_version_description,
+            psc_interface_config=psc_interface_config,
         )
 
         return model, training_id, custom_job_id
@@ -1725,6 +1739,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         predefined_split_column_name: str | None = None,
         timestamp_split_column_name: str | None = None,
         tensorboard: str | None = None,
+        psc_interface_config: PscInterfaceConfig | None = None,
     ) -> CustomContainerTrainingJob:
         """
         Create and submit a Custom Container Training Job pipeline, then exit without waiting for it to complete.
@@ -1985,6 +2000,8 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
                 ``projects/{project}/locations/{location}/tensorboards/{tensorboard}``
                 For more information on configuring your service account please visit:
                 https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training
+        :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+            training.
         """
         self._job = self.get_custom_container_training_job(
             project=project_id,
@@ -2043,6 +2060,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
             model_version_aliases=model_version_aliases,
             model_version_description=model_version_description,
             sync=False,
+            psc_interface_config=psc_interface_config,
         )
         return self._job
 
@@ -2104,6 +2122,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         is_default_version: bool | None = None,
         model_version_aliases: list[str] | None = None,
         model_version_description: str | None = None,
+        psc_interface_config: PscInterfaceConfig | None = None,
     ) -> CustomPythonPackageTrainingJob:
         """
         Create and submit a Custom Python Package Training Job pipeline, then exit without waiting for it to complete.
@@ -2363,6 +2382,8 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
                 ``projects/{project}/locations/{location}/tensorboards/{tensorboard}``
                 For more information on configuring your service account please visit:
                 https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training
+        :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+            training.
         """
         self._job = self.get_custom_python_package_training_job(
             project=project_id,
@@ -2422,6 +2443,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
             model_version_aliases=model_version_aliases,
             model_version_description=model_version_description,
             sync=False,
+            psc_interface_config=psc_interface_config,
         )
 
         return self._job
@@ -2484,6 +2506,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
         predefined_split_column_name: str | None = None,
         timestamp_split_column_name: str | None = None,
         tensorboard: str | None = None,
+        psc_interface_config: PscInterfaceConfig | None = None,
     ) -> CustomTrainingJob:
         """
         Create and submit a Custom Training Job pipeline, then exit without waiting for it to complete.
@@ -2747,6 +2770,8 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
                 ``projects/{project}/locations/{location}/tensorboards/{tensorboard}``
                 For more information on configuring your service account please visit:
                 https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training
+        :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+            training.
         """
         self._job = self.get_custom_training_job(
             project=project_id,
@@ -2806,6 +2831,7 @@ class CustomJobHook(GoogleBaseHook, OperationHelper):
             model_version_aliases=model_version_aliases,
             model_version_description=model_version_description,
             sync=False,
+            psc_interface_config=psc_interface_config,
         )
         return self._job
 

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/src/airflow/providers/google/cloud/hooks/vertex_ai/ray.py

@@ -20,19 +20,10 @@
 from __future__ import annotations
 
 import dataclasses
+from collections.abc import MutableMapping
 from typing import Any
 
-from airflow.exceptions import AirflowOptionalProviderFeatureException
-
-try:
-    import vertex_ray
-    from google._upb._message import ScalarMapContainer  # type: ignore[attr-defined]
-except ImportError:
-    # Fallback for environments where the upb module is not available.
-    raise AirflowOptionalProviderFeatureException(
-        "google._upb._message.ScalarMapContainer is not available. "
-        "Please install the ray package to use this feature."
-    )
+import vertex_ray
 from google.cloud import aiplatform
 from google.cloud.aiplatform.vertex_ray.util import resources
 from google.cloud.aiplatform_v1 import (
@@ -59,7 +50,7 @@ class RayHook(GoogleBaseHook):
         def __encode_value(value: Any) -> Any:
             if isinstance(value, (list, Repeated)):
                 return [__encode_value(nested_value) for nested_value in value]
-            if isinstance(value, ScalarMapContainer):
+            if not isinstance(value, dict) and isinstance(value, MutableMapping):
                 return {key: __encode_value(nested_value) for key, nested_value in dict(value).items()}
             if dataclasses.is_dataclass(value):
                 return dataclasses.asdict(value)

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/src/airflow/providers/google/cloud/operators/vertex_ai/custom_job.py

@@ -51,6 +51,7 @@ if TYPE_CHECKING:
         CustomPythonPackageTrainingJob,
         CustomTrainingJob,
     )
+    from google.cloud.aiplatform_v1.types import PscInterfaceConfig
 
     from airflow.utils.context import Context
 
@@ -110,6 +111,7 @@ class CustomTrainingJobBaseOperator(GoogleCloudBaseOperator):
         predefined_split_column_name: str | None = None,
         timestamp_split_column_name: str | None = None,
         tensorboard: str | None = None,
+        psc_interface_config: PscInterfaceConfig | None = None,
         gcp_conn_id: str = "google_cloud_default",
         impersonation_chain: str | Sequence[str] | None = None,
         **kwargs,
@@ -166,6 +168,7 @@ class CustomTrainingJobBaseOperator(GoogleCloudBaseOperator):
         self.predefined_split_column_name = predefined_split_column_name
         self.timestamp_split_column_name = timestamp_split_column_name
         self.tensorboard = tensorboard
+        self.psc_interface_config = psc_interface_config
         # END Run param
         self.gcp_conn_id = gcp_conn_id
         self.impersonation_chain = impersonation_chain
@@ -473,6 +476,8 @@ class CreateCustomContainerTrainingJobOperator(CustomTrainingJobBaseOperator):
             ``projects/{project}/locations/{location}/tensorboards/{tensorboard}``
             For more information on configuring your service account please visit:
             https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training
+    :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+        training.
     :param gcp_conn_id: The connection ID to use connecting to Google Cloud.
     :param impersonation_chain: Optional service account to impersonate using short-term
         credentials, or chained list of accounts required to get the access_token
@@ -586,6 +591,7 @@ class CreateCustomContainerTrainingJobOperator(CustomTrainingJobBaseOperator):
             timestamp_split_column_name=self.timestamp_split_column_name,
             tensorboard=self.tensorboard,
             sync=True,
+            psc_interface_config=self.psc_interface_config,
         )
 
         if model:
@@ -652,6 +658,7 @@ class CreateCustomContainerTrainingJobOperator(CustomTrainingJobBaseOperator):
             predefined_split_column_name=self.predefined_split_column_name,
             timestamp_split_column_name=self.timestamp_split_column_name,
             tensorboard=self.tensorboard,
+            psc_interface_config=self.psc_interface_config,
         )
         custom_container_training_job_obj.wait_for_resource_creation()
         training_pipeline_id: str = custom_container_training_job_obj.name
@@ -931,6 +938,8 @@ class CreateCustomPythonPackageTrainingJobOperator(CustomTrainingJobBaseOperator
             ``projects/{project}/locations/{location}/tensorboards/{tensorboard}``
             For more information on configuring your service account please visit:
             https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training
+    :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+        training.
     :param gcp_conn_id: The connection ID to use connecting to Google Cloud.
     :param impersonation_chain: Optional service account to impersonate using short-term
         credentials, or chained list of accounts required to get the access_token
@@ -1043,6 +1052,7 @@ class CreateCustomPythonPackageTrainingJobOperator(CustomTrainingJobBaseOperator
             timestamp_split_column_name=self.timestamp_split_column_name,
             tensorboard=self.tensorboard,
             sync=True,
+            psc_interface_config=self.psc_interface_config,
         )
 
         if model:
@@ -1110,6 +1120,7 @@ class CreateCustomPythonPackageTrainingJobOperator(CustomTrainingJobBaseOperator
             predefined_split_column_name=self.predefined_split_column_name,
             timestamp_split_column_name=self.timestamp_split_column_name,
             tensorboard=self.tensorboard,
+            psc_interface_config=self.psc_interface_config,
         )
         custom_python_training_job_obj.wait_for_resource_creation()
         training_pipeline_id: str = custom_python_training_job_obj.name
@@ -1389,6 +1400,8 @@ class CreateCustomTrainingJobOperator(CustomTrainingJobBaseOperator):
             ``projects/{project}/locations/{location}/tensorboards/{tensorboard}``
             For more information on configuring your service account please visit:
             https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-training
+    :param psc_interface_config: Optional. Configuration for Private Service Connect interface used for
+        training.
     :param gcp_conn_id: The connection ID to use connecting to Google Cloud.
     :param impersonation_chain: Optional service account to impersonate using short-term
         credentials, or chained list of accounts required to get the access_token
@@ -1506,6 +1519,7 @@ class CreateCustomTrainingJobOperator(CustomTrainingJobBaseOperator):
             timestamp_split_column_name=self.timestamp_split_column_name,
             tensorboard=self.tensorboard,
             sync=True,
+            psc_interface_config=None,
         )
 
         if model:
@@ -1573,6 +1587,7 @@ class CreateCustomTrainingJobOperator(CustomTrainingJobBaseOperator):
             predefined_split_column_name=self.predefined_split_column_name,
             timestamp_split_column_name=self.timestamp_split_column_name,
             tensorboard=self.tensorboard,
+            psc_interface_config=self.psc_interface_config,
         )
         custom_training_job_obj.wait_for_resource_creation()
         training_pipeline_id: str = custom_training_job_obj.name

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/src/airflow/providers/google/cloud/operators/vertex_ai/dataset.py

@@ -410,7 +410,7 @@ class ImportDataOperator(GoogleCloudBaseOperator, DatasetImportDataResultsCheckH
         )
         initial_dataset_size = self._get_number_of_ds_items(
             dataset=hook.get_dataset(
-                dataset_id=self.dataset_id,
+                dataset=self.dataset_id,
                 project_id=self.project_id,
                 region=self.region,
                 retry=self.retry,
@@ -432,7 +432,7 @@ class ImportDataOperator(GoogleCloudBaseOperator, DatasetImportDataResultsCheckH
         hook.wait_for_operation(timeout=self.timeout, operation=operation)
         result_dataset_size = self._get_number_of_ds_items(
             dataset=hook.get_dataset(
-                dataset_id=self.dataset_id,
+                dataset=self.dataset_id,
                 project_id=self.project_id,
                 region=self.region,
                 retry=self.retry,

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/src/airflow/providers/google/cloud/operators/vertex_ai/ray.py

@@ -282,7 +282,7 @@ class GetRayClusterOperator(RayBaseOperator):
                 location=self.location,
                 cluster_id=self.cluster_id,
             )
-            self.log.info("Cluster was gotten.")
+            self.log.info("Cluster data has been retrieved.")
             ray_cluster_dict = self.hook.serialize_cluster_obj(ray_cluster)
             return ray_cluster_dict
         except NotFound as not_found_err:

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/src/airflow/providers/google/cloud/transfers/bigquery_to_postgres.py

@@ -21,6 +21,9 @@ from __future__ import annotations
 
 from typing import TYPE_CHECKING
 
+from psycopg2.extensions import register_adapter
+from psycopg2.extras import Json
+
 from airflow.providers.google.cloud.hooks.bigquery import BigQueryHook
 from airflow.providers.google.cloud.transfers.bigquery_to_sql import BigQueryToSqlBaseOperator
 from airflow.providers.google.cloud.utils.bigquery_get_data import bigquery_get_data
@@ -76,6 +79,8 @@ class BigQueryToPostgresOperator(BigQueryToSqlBaseOperator):
         self.replace_index = replace_index
 
     def get_sql_hook(self) -> PostgresHook:
+        register_adapter(list, Json)
+        register_adapter(dict, Json)
         return PostgresHook(database=self.database, postgres_conn_id=self.postgres_conn_id)
 
     def execute(self, context: Context) -> None:

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/tests/system/google/cloud/dataproc_metastore/example_dataproc_metastore.py

@@ -52,7 +52,7 @@ ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID", "default")
 SERVICE_ID = f"{DAG_ID}-service-{ENV_ID}".replace("_", "-")
 METADATA_IMPORT_ID = f"{DAG_ID}-metadata-{ENV_ID}".replace("_", "-")
 
-REGION = "europe-west1"
+REGION = "us-central1"
 RESOURCE_DATA_BUCKET = "airflow-system-tests-resources"
 BUCKET_NAME = f"bucket_{DAG_ID}_{ENV_ID}"
 TIMEOUT = 2400

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/tests/system/google/cloud/dataproc_metastore/example_dataproc_metastore_backup.py

@@ -45,8 +45,8 @@ ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID", "default")
 
 SERVICE_ID = f"{DAG_ID}-service-{ENV_ID}".replace("_", "-")
 BACKUP_ID = f"{DAG_ID}-backup-{ENV_ID}".replace("_", "-")
-REGION = "europe-west1"
-TIMEOUT = 1200
+REGION = "europe-west3"
+TIMEOUT = 2400
 # Service definition
 SERVICE = {
     "name": "test-service",

{apache_airflow_providers_google-17.0.0rc1 → apache_airflow_providers_google-17.1.0rc1}/tests/system/google/cloud/workflows/example_workflows.py

@@ -77,7 +77,7 @@ SLEEP_WORKFLOW_CONTENT = """
 - someSleep:
     call: sys.sleep
     args:
-        seconds: 120
+        seconds: 200
 """
 
 SLEEP_WORKFLOW = {