databricks-sdk 0.45.0__tar.gz → 0.47.0__tar.gz

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/CHANGELOG.md

@@ -1,5 +1,61 @@
 # Version changelog
 
+## Release v0.47.0
+
+### Bug Fixes
+
+ * Ensure that refresh tokens are returned when using the `external-browser` credentials strategy.
+
+### API Changes
+* Added `abfss`, `dbfs`, `error_message`, `execution_duration_seconds`, `file`, `gcs`, `s3`, `status`, `volumes` and `workspace` fields for `databricks.sdk.service.compute.InitScriptInfoAndExecutionDetails`.
+* [Breaking] Added `forecast_granularity` field for `databricks.sdk.service.ml.CreateForecastingExperimentRequest`.
+* Added `jwks_uri` field for `databricks.sdk.service.oauth2.OidcFederationPolicy`.
+* Added `fallback_config` field for `databricks.sdk.service.serving.AiGatewayConfig`.
+* Added `custom_provider_config` field for `databricks.sdk.service.serving.ExternalModel`.
+* Added `fallback_config` field for `databricks.sdk.service.serving.PutAiGatewayRequest`.
+* Added `fallback_config` field for `databricks.sdk.service.serving.PutAiGatewayResponse`.
+* Added `aliases`, `comment`, `data_type`, `dependency_list`, `full_data_type`, `id`, `input_params`, `name`, `properties`, `routine_definition`, `schema`, `securable_kind`, `share`, `share_id`, `storage_location` and `tags` fields for `databricks.sdk.service.sharing.DeltaSharingFunction`.
+* Added `access_token_failure`, `allocation_timeout`, `allocation_timeout_node_daemon_not_ready`, `allocation_timeout_no_healthy_clusters`, `allocation_timeout_no_matched_clusters`, `allocation_timeout_no_ready_clusters`, `allocation_timeout_no_unallocated_clusters`, `allocation_timeout_no_warmed_up_clusters`, `aws_inaccessible_kms_key_failure`, `aws_instance_profile_update_failure`, `aws_invalid_key_pair`, `aws_invalid_kms_key_state`, `aws_resource_quota_exceeded`, `azure_packed_deployment_partial_failure`, `bootstrap_timeout_due_to_misconfig`, `budget_policy_limit_enforcement_activated`, `budget_policy_resolution_failure`, `cloud_account_setup_failure`, `cloud_operation_cancelled`, `cloud_provider_instance_not_launched`, `cloud_provider_launch_failure_due_to_misconfig`, `cloud_provider_resource_stockout_due_to_misconfig`, `cluster_operation_throttled`, `cluster_operation_timeout`, `control_plane_request_failure_due_to_misconfig`, `data_access_config_changed`, `disaster_recovery_replication`, `driver_eviction`, `driver_launch_timeout`, `driver_node_unreachable`, `driver_out_of_disk`, `driver_out_of_memory`, `driver_pod_creation_failure`, `driver_unexpected_failure`, `dynamic_spark_conf_size_exceeded`, `eos_spark_image`, `executor_pod_unscheduled`, `gcp_api_rate_quota_exceeded`, `gcp_forbidden`, `gcp_iam_timeout`, `gcp_inaccessible_kms_key_failure`, `gcp_insufficient_capacity`, `gcp_ip_space_exhausted`, `gcp_kms_key_permission_denied`, `gcp_not_found`, `gcp_resource_quota_exceeded`, `gcp_service_account_access_denied`, `gcp_service_account_not_found`, `gcp_subnet_not_ready`, `gcp_trusted_image_projects_violated`, `gke_based_cluster_termination`, `init_container_not_finished`, `instance_pool_max_capacity_reached`, `instance_pool_not_found`, `instance_unreachable_due_to_misconfig`, `internal_capacity_failure`, `invalid_aws_parameter`, `invalid_instance_placement_protocol`, `invalid_worker_image_failure`, `in_penalty_box`, `lazy_allocation_timeout`, `maintenance_mode`, `netvisor_setup_timeout`, `no_matched_k8s`, `no_matched_k8s_testing_tag`, `pod_assignment_failure`, `pod_scheduling_failure`, `resource_usage_blocked`, `secret_creation_failure`, `serverless_long_running_terminated`, `spark_image_download_throttled`, `spark_image_not_found`, `ssh_bootstrap_failure`, `storage_download_failure_due_to_misconfig`, `storage_download_failure_slow`, `storage_download_failure_throttled`, `unexpected_pod_recreation`, `user_initiated_vm_termination` and `workspace_update` enum values for `databricks.sdk.service.compute.TerminationReasonCode`.
+* Added `generated_sql_query_too_long_exception` and `missing_sql_query_exception` enum values for `databricks.sdk.service.dashboards.MessageErrorType`.
+* Added `balanced` enum value for `databricks.sdk.service.jobs.PerformanceTarget`.
+* Added `listing_resource` enum value for `databricks.sdk.service.marketplace.FileParentType`.
+* Added `app` enum value for `databricks.sdk.service.marketplace.MarketplaceFileType`.
+* Added `custom` enum value for `databricks.sdk.service.serving.ExternalModelProvider`.
+* [Breaking] Changed `create_experiment()` method for [w.forecasting](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/ml/forecasting.html) workspace-level service with new required argument order.
+* Changed `instance_type_id` field for `databricks.sdk.service.compute.NodeInstanceType` to be required.
+* Changed `category` field for `databricks.sdk.service.compute.NodeType` to be required.
+* [Breaking] Changed `functions` field for `databricks.sdk.service.sharing.ListProviderShareAssetsResponse` to type `databricks.sdk.service.sharing.DeltaSharingFunctionList` dataclass.
+* [Breaking] Changed waiter for [ClustersAPI.create](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.create) method.
+* [Breaking] Changed waiter for [ClustersAPI.delete](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.delete) method.
+* [Breaking] Changed waiter for [ClustersAPI.edit](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.edit) method.
+* [Breaking] Changed waiter for [ClustersAPI.get](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.get) method.
+* [Breaking] Changed waiter for [ClustersAPI.resize](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.resize) method.
+* [Breaking] Changed waiter for [ClustersAPI.restart](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.restart) method.
+* [Breaking] Changed waiter for [ClustersAPI.start](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.start) method.
+* [Breaking] Changed waiter for [ClustersAPI.update](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/compute/clusters.html#databricks.sdk.service.compute.ClustersAPI.update) method.
+* [Breaking] Removed `execution_details` and `script` fields for `databricks.sdk.service.compute.InitScriptInfoAndExecutionDetails`.
+* [Breaking] Removed `supports_elastic_disk` field for `databricks.sdk.service.compute.NodeType`.
+* [Breaking] Removed `data_granularity_quantity` and `data_granularity_unit` fields for `databricks.sdk.service.ml.CreateForecastingExperimentRequest`.
+* [Breaking] Removed `aliases`, `comment`, `data_type`, `dependency_list`, `full_data_type`, `id`, `input_params`, `name`, `properties`, `routine_definition`, `schema`, `securable_kind`, `share`, `share_id`, `storage_location` and `tags` fields for `databricks.sdk.service.sharing.Function`.
+
+
+## Release v0.46.0
+
+### New Features and Improvements
+* [Experimental] Add support for async token refresh ([#916](https://github.com/databricks/databricks-sdk-py/pull/916)).
+  This can be enabled with by setting the following setting:
+  ```
+  export DATABRICKS_ENABLE_EXPERIMENTAL_ASYNC_TOKEN_REFRESH=1.
+  ```
+  This feature and its setting are experimental and may be removed in future releases.
+
+### API Changes
+* Added [w.forecasting](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/ml/forecasting.html) workspace-level service.
+* Added `statement_id` field for `databricks.sdk.service.dashboards.GenieQueryAttachment`.
+* Added `could_not_get_model_deployments_exception` enum value for `databricks.sdk.service.dashboards.MessageErrorType`.
+* [Breaking] Removed `jwks_uri` field for `databricks.sdk.service.oauth2.OidcFederationPolicy`.
+
+
 ## Release v0.45.0
 
 ### New Features and Improvements

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: databricks-sdk
-Version: 0.45.0
+Version: 0.47.0
 Summary: Databricks SDK for Python (Beta)
 Project-URL: Documentation, https://databricks-sdk-py.readthedocs.io
 Keywords: databricks,sdk
@@ -44,12 +44,13 @@ Requires-Dist: langchain-openai; python_version > "3.7" and extra == "dev"
 Requires-Dist: httpx; extra == "dev"
 Requires-Dist: build; extra == "dev"
 Provides-Extra: notebook
-Requires-Dist: ipython<9,>=8; extra == "notebook"
+Requires-Dist: ipython<10,>=8; extra == "notebook"
 Requires-Dist: ipywidgets<9,>=8; extra == "notebook"
 Provides-Extra: openai
 Requires-Dist: openai; extra == "openai"
 Requires-Dist: langchain-openai; python_version > "3.7" and extra == "openai"
 Requires-Dist: httpx; extra == "openai"
+Dynamic: license-file
 
 # Databricks SDK for Python (Beta)
 

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/databricks/sdk/__init__.py

@@ -8,6 +8,7 @@ import databricks.sdk.dbutils as dbutils
 import databricks.sdk.service as service
 from databricks.sdk import azure
 from databricks.sdk.credentials_provider import CredentialsStrategy
+from databricks.sdk.data_plane import DataPlaneTokenSource
 from databricks.sdk.mixins.compute import ClustersExt
 from databricks.sdk.mixins.files import DbfsExt, FilesExt
 from databricks.sdk.mixins.jobs import JobsExt
@@ -63,7 +64,8 @@ from databricks.sdk.service.marketplace import (
     ProviderExchangeFiltersAPI, ProviderExchangesAPI, ProviderFilesAPI,
     ProviderListingsAPI, ProviderPersonalizationRequestsAPI,
     ProviderProviderAnalyticsDashboardsAPI, ProviderProvidersAPI)
-from databricks.sdk.service.ml import ExperimentsAPI, ModelRegistryAPI
+from databricks.sdk.service.ml import (ExperimentsAPI, ForecastingAPI,
+                                       ModelRegistryAPI)
 from databricks.sdk.service.oauth2 import (AccountFederationPolicyAPI,
                                            CustomAppIntegrationAPI,
                                            OAuthPublishedAppsAPI,
@@ -284,8 +286,11 @@ class WorkspaceClient:
         self._secrets = service.workspace.SecretsAPI(self._api_client)
         self._service_principals = service.iam.ServicePrincipalsAPI(self._api_client)
         self._serving_endpoints = serving_endpoints
+        serving_endpoints_data_plane_token_source = DataPlaneTokenSource(
+            self._config.host, self._config.oauth_token, not self._config.enable_experimental_async_token_refresh
+        )
         self._serving_endpoints_data_plane = service.serving.ServingEndpointsDataPlaneAPI(
-            self._api_client, serving_endpoints
+            self._api_client, serving_endpoints, serving_endpoints_data_plane_token_source
         )
         self._settings = service.settings.SettingsAPI(self._api_client)
         self._shares = service.sharing.SharesAPI(self._api_client)
@@ -305,6 +310,7 @@ class WorkspaceClient:
         self._workspace = WorkspaceExt(self._api_client)
         self._workspace_bindings = service.catalog.WorkspaceBindingsAPI(self._api_client)
         self._workspace_conf = service.settings.WorkspaceConfAPI(self._api_client)
+        self._forecasting = service.ml.ForecastingAPI(self._api_client)
 
     @property
     def config(self) -> client.Config:
@@ -808,6 +814,11 @@ class WorkspaceClient:
         """This API allows updating known workspace settings for advanced users."""
         return self._workspace_conf
 
+    @property
+    def forecasting(self) -> service.ml.ForecastingAPI:
+        """The Forecasting API allows you to create and get serverless forecasting experiments."""
+        return self._forecasting
+
     def get_workspace_id(self) -> int:
         """Get the workspace ID of the workspace that this client is connected to."""
         response = self._api_client.do("GET", "/api/2.0/preview/scim/v2/Me", response_headers=["X-Databricks-Org-Id"])

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/databricks/sdk/config.py

@@ -95,6 +95,10 @@ class Config:
     max_connections_per_pool: int = ConfigAttribute()
     databricks_environment: Optional[DatabricksEnvironment] = None
 
+    enable_experimental_async_token_refresh: bool = ConfigAttribute(
+        env="DATABRICKS_ENABLE_EXPERIMENTAL_ASYNC_TOKEN_REFRESH"
+    )
+
     enable_experimental_files_api_client: bool = ConfigAttribute(env="DATABRICKS_ENABLE_EXPERIMENTAL_FILES_API_CLIENT")
     files_api_client_download_max_total_recovers = None
     files_api_client_download_max_total_recovers_without_progressing = 1

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/databricks/sdk/credentials_provider.py

@@ -191,6 +191,7 @@ def oauth_service_principal(cfg: "Config") -> Optional[CredentialsProvider]:
         token_url=oidc.token_endpoint,
         scopes=["all-apis"],
         use_header=True,
+        disable_async=not cfg.enable_experimental_async_token_refresh,
     )
 
     def inner() -> Dict[str, str]:
@@ -290,6 +291,7 @@ def azure_service_principal(cfg: "Config") -> CredentialsProvider:
             token_url=f"{aad_endpoint}{cfg.azure_tenant_id}/oauth2/token",
             endpoint_params={"resource": resource},
             use_params=True,
+            disable_async=not cfg.enable_experimental_async_token_refresh,
         )
 
     _ensure_host_present(cfg, token_source_for)
@@ -355,6 +357,7 @@ def github_oidc_azure(cfg: "Config") -> Optional[CredentialsProvider]:
         token_url=f"{aad_endpoint}{cfg.azure_tenant_id}/oauth2/token",
         endpoint_params=params,
         use_params=True,
+        disable_async=not cfg.enable_experimental_async_token_refresh,
     )
 
     def refreshed_headers() -> Dict[str, str]:
@@ -458,8 +461,9 @@ class CliTokenSource(Refreshable):
         token_type_field: str,
         access_token_field: str,
         expiry_field: str,
+        disable_async: bool = True,
     ):
-        super().__init__()
+        super().__init__(disable_async=disable_async)
         self._cmd = cmd
         self._token_type_field = token_type_field
         self._access_token_field = access_token_field
@@ -690,6 +694,7 @@ class DatabricksCliTokenSource(CliTokenSource):
             token_type_field="token_type",
             access_token_field="access_token",
             expiry_field="expiry",
+            disable_async=not cfg.enable_experimental_async_token_refresh,
         )
 
     @staticmethod

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/databricks/sdk/data_plane.py

@@ -2,7 +2,7 @@ from __future__ import annotations
 
 import threading
 from dataclasses import dataclass
-from typing import Callable, List, Optional
+from typing import Callable, Optional
 from urllib import parse
 
 from databricks.sdk import oauth
@@ -88,61 +88,3 @@ class DataPlaneDetails:
     """URL used to query the endpoint through the DataPlane."""
     token: Token
     """Token to query the DataPlane endpoint."""
-
-
-## Old implementation. #TODO: Remove after the new implementation is used
-
-
-class DataPlaneService:
-    """Helper class to fetch and manage DataPlane details."""
-
-    from .service.serving import DataPlaneInfo
-
-    def __init__(self):
-        self._data_plane_info = {}
-        self._tokens = {}
-        self._lock = threading.Lock()
-
-    def get_data_plane_details(
-        self,
-        method: str,
-        params: List[str],
-        info_getter: Callable[[], DataPlaneInfo],
-        refresh: Callable[[str], Token],
-    ):
-        """Get and cache information required to query a Data Plane endpoint using the provided methods.
-
-        Returns a cached DataPlaneDetails if the details have already been fetched previously and are still valid.
-        If not, it uses the provided functions to fetch the details.
-
-        :param method: method name. Used to construct a unique key for the cache.
-        :param params: path params used in the "get" operation which uniquely determine the object. Used to construct a unique key for the cache.
-        :param info_getter: function which returns the DataPlaneInfo. It will only be called if the information is not already present in the cache.
-        :param refresh: function to refresh the token. It will only be called if the token is missing or expired.
-        """
-        all_elements = params.copy()
-        all_elements.insert(0, method)
-        map_key = "/".join(all_elements)
-        info = self._data_plane_info.get(map_key)
-        if not info:
-            self._lock.acquire()
-            try:
-                info = self._data_plane_info.get(map_key)
-                if not info:
-                    info = info_getter()
-                    self._data_plane_info[map_key] = info
-            finally:
-                self._lock.release()
-
-        token = self._tokens.get(map_key)
-        if not token or not token.valid:
-            self._lock.acquire()
-            token = self._tokens.get(map_key)
-            try:
-                if not token or not token.valid:
-                    token = refresh(info.authorization_details)
-                    self._tokens[map_key] = token
-            finally:
-                self._lock.release()
-
-        return DataPlaneDetails(endpoint_url=info.endpoint_url, token=token)

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/databricks/sdk/oauth.py

@@ -426,12 +426,16 @@ class SessionCredentials(Refreshable):
         client_id: str,
         client_secret: str = None,
         redirect_url: str = None,
+        disable_async: bool = True,
     ):
         self._token_endpoint = token_endpoint
         self._client_id = client_id
         self._client_secret = client_secret
         self._redirect_url = redirect_url
-        super().__init__(token)
+        super().__init__(
+            token=token,
+            disable_async=disable_async,
+        )
 
     def as_dict(self) -> dict:
         return {"token": self.token().as_dict()}
@@ -625,7 +629,11 @@ class OAuthClient:
     ):
 
         if not scopes:
-            scopes = ["all-apis"]
+            # all-apis ensures that the returned OAuth token can be used with all APIs, aside
+            # from direct-to-dataplane APIs.
+            # offline_access ensures that the response from the Authorization server includes
+            # a refresh token.
+            scopes = ["all-apis", "offline_access"]
 
         self.redirect_url = redirect_url
         self._client_id = client_id
@@ -650,8 +658,6 @@ class OAuthClient:
             return lambda: {}
 
         config = Config(host=host, credentials_strategy=noop_credentials)
-        if not scopes:
-            scopes = ["all-apis"]
         oidc = config.oidc_endpoints
         if not oidc:
             raise ValueError(f"{host} does not support OAuth")
@@ -708,9 +714,10 @@ class ClientCredentials(Refreshable):
     scopes: List[str] = None
     use_params: bool = False
     use_header: bool = False
+    disable_async: bool = True
 
     def __post_init__(self):
-        super().__init__()
+        super().__init__(disable_async=self.disable_async)
 
     def refresh(self) -> Token:
         params = {"grant_type": "client_credentials"}

{databricks_sdk-0.45.0 → databricks_sdk-0.47.0}/databricks/sdk/service/catalog.py

@@ -9471,6 +9471,8 @@ class UpdateWorkspaceBindingsParameters:
 
 @dataclass
 class ValidateCredentialRequest:
+    """Next ID: 17"""
+
     aws_iam_role: Optional[AwsIamRole] = None
     """The AWS IAM role configuration"""