fsspec-databricks 0.2.0__tar.gz → 0.3.0__tar.gz

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/.gitignore

@@ -82,9 +82,6 @@ ENV/
 # vi*
 *.swp
 
-# UV
-uv.lock
-
 build/
 downstream/*
 

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/DEVELOPMENT.md

@@ -105,21 +105,63 @@ The workflows are in `.github/workflows/`:
 
 Each run's work directories are `<root>/<branch>/<Python version>` under the environment's roots.
 
+### Supply chain
+
+The test jobs hold the workspaces' credentials, so what they run is fixed:
+
+- `uv.lock` is committed, with every package's hash. CI installs with `uv sync --locked`, which fails if the lock is
+  out of date.
+- Actions are pinned to commit SHAs. setup-uv installs only uv versions whose checksums it knows (`latest-known`).
+- Dependabot updates `uv.lock` and the Actions' SHAs weekly.
+- Workflows run only on pushes and releases, never on pull requests from forks.
+- Each workflow's token can only read the repository. Publishing adds `id-token: write`, for PyPI's trusted
+  publishing.
+- The release build uses no cache.
+
+ruff, run with `uvx`, is the exception: CI runs its latest version, in a job without secrets.
+
 ### Environments
 
 Each GitHub environment is a Databricks workspace. Create at least `default`, and `azure`, `aws` or `gcp` as needed.
 The tests authenticate as a service principal with read-write access to the work directories.
 
-| Secret                     | Value                                         |
-|----------------------------|-----------------------------------------------|
-| `DATABRICKS_HOST`          | The workspace's URL                           |
-| `DATABRICKS_CLIENT_ID`     | The service principal's client ID             |
-| `DATABRICKS_CLIENT_SECRET` | The service principal's client secret         |
+Every environment has these:
+
+| Name                                               | Value                                          |
+|----------------------------------------------------|------------------------------------------------|
+| `DATABRICKS_HOST` (secret)                         | The workspace's URL                            |
+| `FSSPEC_DATABRICKS_VOLUME_TEST_ROOT` (variable)    | The POSIX path of the Volume work directory    |
+| `FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT` (variable) | The POSIX path of the Workspace work directory |
+
+The variable `DATABRICKS_AUTH_TYPE` chooses how the tests authenticate:
+
+| `DATABRICKS_AUTH_TYPE` | Authentication                                        | Also needs                                                       |
+|------------------------|-------------------------------------------------------|------------------------------------------------------------------|
+| Not set                | The service principal's client secret                 | `DATABRICKS_CLIENT_ID` and `DATABRICKS_CLIENT_SECRET` (secrets)  |
+| `github-oidc`          | Databricks federation policy                          | `DATABRICKS_CLIENT_ID` (secret), `DATABRICKS_TOKEN_AUDIENCE`     |
+| `github-oidc-azure`    | Federated credential of a Microsoft Entra ID app      | `ARM_CLIENT_ID`, the app's ID                                    |
+
+Prefer federation: it needs no stored secret. GitHub's OIDC token identifies each environment by the subject
+`repo:sugibuchi/fsspec-databricks:environment:<name>`. The federation's trust must name that subject and the issuer
+`https://token.actions.githubusercontent.com`.
+
+- **`github-oidc`**: a federation policy on the service principal, in the Databricks account console. Its audience is
+  `DATABRICKS_TOKEN_AUDIENCE`, e.g. the account ID.
+  The SDK's `github-oidc` exchanges a new token for each request, which slows the tests down.
+  So `.github/scripts/with-id-token-file.sh` keeps GitHub's token in a file and runs the tests with `file-oidc`, which
+  caches the exchanged token.
+- **`github-oidc-azure`**: a federated credential on the Entra ID app, with the audience `api://AzureADTokenExchange`.
+  The app is the workspace's service principal. This needs no access to the account console.
+
+With federation, the client secret is not passed to the tests, even if the environment still has one.
+
+The environments are:
 
-| Variable                                | Value                                         |
-|-----------------------------------------|-----------------------------------------------|
-| `FSSPEC_DATABRICKS_VOLUME_TEST_ROOT`    | The POSIX path of the Volume work directory    |
-| `FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT` | The POSIX path of the Workspace work directory |
+| Environment | Workspace                        | Authentication      |
+|-------------|----------------------------------|---------------------|
+| `default`   | Databricks Free Edition          | Client secret       |
+| `azure`     | Azure Databricks                 | `github-oidc-azure` |
+| `gcp`       | Databricks on Google Cloud       | `github-oidc`       |
 
 At the repository level:
 
@@ -138,6 +180,12 @@ The version comes from the git tag, through hatch-vcs.
 
 `uv publish` authenticates to PyPI with the workflow's OIDC token, through trusted publishing.
 
+Repository rules protect the release path:
+
+- The ruleset "Release tags" lets only repository admins create, move or delete `v*` tags.
+- The `pypi` environment deploys only from `v*` tags.
+- The ruleset "Default branch" blocks force-pushes to `main` and its deletion.
+
 ## Design
 
 ### One file system for all of Databricks' paths
@@ -171,14 +219,19 @@ exception.
 
 ### Picklable
 
-File systems are pickled to run in other processes, e.g. by Dask. Clients, event loops, threads and sessions are left
-out, and made again on first use.
+File systems are pickled to run in other processes, e.g. by Dask. fsspec pickles a file system as its constructor
+arguments, and unpickling calls the constructor again:
+
+- clients, event loops, threads and sessions are not pickled, and are made again on first use;
+- attributes set after construction are lost;
+- every constructor argument must be picklable.
 
-A `WorkspaceClient` given as `client` is kept, and is not picklable. Other ways of authenticating are.
+A `WorkspaceClient` given as `client` is not picklable. Other ways of authenticating are.
 
 ### Known limits
 
-- **Multipart upload uses an undocumented API.** Databricks may change it without notice.
+- **Multipart upload is not in the REST reference.** The Databricks SDK for Python uses the same endpoints, as an
+  experimental feature, in `databricks/sdk/mixins/files.py`. Compare with it when the API changes.
 - **Workspace files are at most 10 MB.** The Workspace API's limit.
 - **No append or update modes on Volumes.** `ab` and `r+b` are refused.
 - **No listing cache.** Each call reaches the workspace.
@@ -225,7 +278,6 @@ Within one backend, both go to the backend.
 
 - the Databricks SDK `Config` and `WorkspaceClient`, made on first use from `client`, `config` or `config_params`;
 - `close()`, also called at exit and on garbage collection;
-- pickling without the client and config;
 - the `dbfs` protocol: `_strip_protocol` and `unstrip_protocol` use `path.py`.
 
 Its `mkdir`, `makedirs`, `rmdir` and `_rm` only check preconditions, such as a missing parent or a file where a folder
@@ -263,20 +315,48 @@ when no glob pattern is involved. Files are opened with the SDK's `dbfs.open`.
 
 ### Volumes
 
-`VolumeFileSystem`, in `volume.py`, subclasses `DBFS`, as the DBFS API also lists Volume folders.
+`VolumeFileSystem`, in `volume.py`, subclasses `DBFS`. It keeps the SDK's `dbfs` for `exists` and for creating
+folders. Listings, copies, moves and deletes inside a volume use the Files API and the backend's own file objects.
 
 The APIs called depend on the depth of the path:
 
-| Path                                   | `info`                                                    | `ls`            |
-|----------------------------------------|-----------------------------------------------------------|-----------------|
-| `/Volumes`                             | None: a fixed folder                                      | `catalogs.list` |
-| `/Volumes/<catalog>`                   | `catalogs.get`                                            | `schemas.list`  |
-| `/Volumes/<catalog>/<schema>`          | `schemas.get`                                             | `volumes.list`  |
-| `/Volumes/<catalog>/<schema>/<volume>` | `volumes.read`                                            | `dbfs.list`     |
-| Deeper                                 | `files.get_directory_metadata`, else `files.get_metadata` | `dbfs.list`     |
+| Path                                   | `info`                                                    | `ls`                            |
+|----------------------------------------|-----------------------------------------------------------|---------------------------------|
+| `/Volumes`                             | None: a fixed folder                                      | `catalogs.list`                 |
+| `/Volumes/<catalog>`                   | `catalogs.get`                                            | `schemas.list`                  |
+| `/Volumes/<catalog>/<schema>`          | `schemas.get`                                             | `volumes.list`                  |
+| `/Volumes/<catalog>/<schema>/<volume>` | `volumes.read`                                            | `files.list_directory_contents` |
+| Deeper                                 | `files.get_metadata`, else `files.get_directory_metadata` | `files.list_directory_contents` |
+
+`ls` of a file returns the file itself, from `files.get_metadata` after the listing fails.
 
 `include_browse` is passed to the Unity Catalog APIs.
 
+Each metadata call is a round trip, so the checks inside a volume make as few as they can:
+
+| Check                                         | Calls                                           |
+|-----------------------------------------------|-------------------------------------------------|
+| `info` of a file, so also `size` and reading  | 1: `files.get_metadata`                         |
+| `info` of a directory                         | 2: `files.get_metadata`, then the directory's   |
+| `isdir`, so also `mkdir` and `rmdir`          | 1: `files.get_directory_metadata`               |
+| `isfile`                                      | 1: `files.get_metadata`                         |
+| Opening for writing                           | 1: `isdir`, to refuse directories               |
+| `ls` of a directory                           | 1: `files.list_directory_contents`              |
+
+`DatabricksFileSystem` passes `isdir` and `isfile` to the backends. `test_volume_fs_metadata_calls` counts the calls.
+
+Copies, moves and deletes inside a volume:
+
+- `cp_file` streams the file from a `VolumeReadableFile` into a `VolumeWritableFile`, as the Files API has no copy. The
+  SDK's `dbfs.copy` would hold the whole file in memory.
+- `mv` is fsspec's default: a copy, then a delete.
+- `rm` of a file checks it with `info`, then calls `files.delete`.
+- A recursive `rm` lists the tree one depth at a time, deletes the files, then the folders from the deepest. Each step
+  runs up to `_max_delete_concurrency` requests in a thread pool.
+- Paths of volumes, schemas and catalogs go to `DBFS`.
+
+`test_volume_fs_rm_calls` and `test_volume_fs_cp_file_streams` check these on a fake `WorkspaceClient.files`.
+
 #### The IO event loop
 
 Each `VolumeFileSystem` runs an asyncio event loop in a daemon thread, `dbfs-io-thread`, started on first use. It
@@ -312,6 +392,9 @@ loop:
 `_compute_block_size` gives the block size: the smallest power-of-2 multiple of the minimum that covers the target with
 `max_concurrency` blocks, capped at the maximum.
 
+A range request that gets a `429`, `500`, `502`, `503` or `504` is retried up to 3 times, with exponential backoff and
+jitter. Writes are not retried: a repeated `PUT` can fail with "already exists" when the first one went through.
+
 #### Writing
 
 `VolumeWritableFile` subclasses `AbstractAsyncWritableFile`. Writes are buffered:

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: fsspec-databricks
-Version: 0.2.0
+Version: 0.3.0
 Summary: `fsspec`-based file system interface for Databricks file system"s"
 Project-URL: Homepage, https://github.com/sugibuchi/fsspec-databricks
 Author-email: Tsuyoshi SUGIBUCHI <sugibuchi@gmail.com>
@@ -117,7 +117,8 @@ to Workspace files only in workspaces with legacy DBFS disabled. See
 
 **Early beta.** The API and behaviour may change.
 
-* Writing Volume files uses Databricks' multipart upload API, which is undocumented and may change without notice.
+* Writing Volume files uses the Files API's multipart upload. The Databricks REST reference does not document it, but
+  the official Databricks SDK for Python uses it, as an experimental feature.
 * Databricks on AWS is not tested.
 * Legacy DBFS is not tested.
 

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/README.md

@@ -92,7 +92,8 @@ to Workspace files only in workspaces with legacy DBFS disabled. See
 
 **Early beta.** The API and behaviour may change.
 
-* Writing Volume files uses Databricks' multipart upload API, which is undocumented and may change without notice.
+* Writing Volume files uses the Files API's multipart upload. The Databricks REST reference does not document it, but
+  the official Databricks SDK for Python uses it, as an experimental feature.
 * Databricks on AWS is not tested.
 * Legacy DBFS is not tested.
 

fsspec_databricks-0.3.0/SECURITY.md

@@ -0,0 +1,30 @@
+# Security policy
+
+## Supported versions
+
+Security fixes are made in the latest release only.
+
+## Reporting a vulnerability
+
+Report vulnerabilities privately, through
+[GitHub's private vulnerability reporting](https://github.com/sugibuchi/fsspec-databricks/security/advisories/new).
+Do not open a public issue.
+
+Include:
+
+- the version of `fsspec-databricks`, and of `databricks-sdk`;
+- the steps to reproduce, or a proof of concept;
+- the impact, e.g. leaked credentials or access to files beyond the user's permissions.
+
+Never include real credentials, tokens or workspace URLs in a report.
+
+## Scope
+
+`fsspec-databricks` accesses Databricks with the credentials of its user, through the Databricks SDK. In scope are,
+for example:
+
+- credentials or tokens written to logs, exceptions or pickled file systems;
+- access to files or paths beyond what the given path names;
+- requests sent to hosts other than the workspace, its storage proxy, or the URLs Databricks returns.
+
+Vulnerabilities of Databricks itself or of the Databricks SDK go to Databricks.

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/USAGE.md

@@ -166,6 +166,14 @@ fs.ls("dbfs:/Volumes/my_catalog/my_schema/my_volume/data")
 Reads and writes run concurrent requests. Large writes use multipart upload, or resumable upload where the workspace
 requires it.
 
+Reads retry after transient server errors. Writes do not.
+
+Copies, moves and deletes:
+
+- A copy reads the file and writes it again, as the Files API cannot copy. It uses the options for reads and writes.
+- A move is a copy, then a delete. It replaces an existing file at the destination.
+- A recursive delete sends concurrent requests.
+
 ### Modes
 
 | Mode  | Supported | Notes                            |

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/pyproject.toml

@@ -73,6 +73,15 @@ source = [
 source = "vcs"
 raw-options = { 'version_scheme' = 'post-release' }
 
+[tool.hatch.build.targets.sdist]
+# Files for development only
+exclude = [
+    ".github",
+    ".pre-commit-config.yaml",
+    "CLAUDE.md",
+    "uv.lock",
+]
+
 [tool.ruff]
 exclude = [
     ".tox",

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/src/fsspec_databricks/base.py

@@ -90,17 +90,6 @@ class AbstractDatabricksFileSystem(AbstractFileSystem):
         _register_cleanup(self)
         self.log.debug("Opened: %s", self)
 
-    def __getstate__(self):
-        state = self.__dict__.copy()
-        del state["_AbstractDatabricksFileSystem__client"]
-        del state["_AbstractDatabricksFileSystem__config"]
-        return state
-
-    def __setstate__(self, state):
-        self.__dict__.update(state)
-        self.__client = None
-        self.__config = None
-
     def close(self):
         """Close the file system and release any resources."""
         if not self.closed:
@@ -197,6 +186,16 @@ class AbstractDatabricksFileSystem(AbstractFileSystem):
     def cp_file(self, path1, path2, **kwargs):
         self.log.debug("Copying file: path1=%s, path2=%s", path1, path2)
 
+    def _cp_directory(self, path2):
+        """Copy a directory: create the destination, as directories have no content."""
+        try:
+            info = self.info(path2)
+        except FileNotFoundError:
+            self.mkdirs(path2)
+            return
+        if info["type"] != "directory":
+            raise file_exists_error(path2)
+
     def _rm(self, path):
         # Implement only the precondition check here
         # Actual operations must be implemented by subclasses
@@ -213,15 +212,17 @@ class AbstractDatabricksFileSystem(AbstractFileSystem):
             "Making directory: path=%s, create_parents=%s", path, create_parents
         )
 
-        try:
-            info = self.info(self._parent(path))
-            if info["type"] != "directory":
-                raise not_a_directory_error(self._parent(path))
-        except FileNotFoundError:
-            if create_parents:
-                return
-            else:
-                raise
+        parent = self._parent(path)
+        if not self.isdir(parent):
+            # Tell a missing parent from a file
+            try:
+                self.info(parent)
+            except FileNotFoundError:
+                if create_parents:
+                    return
+                else:
+                    raise
+            raise not_a_directory_error(parent)
 
         if self.exists(path):
             raise file_exists_error(path)
@@ -248,8 +249,9 @@ class AbstractDatabricksFileSystem(AbstractFileSystem):
         # Actual operations must be implemented by subclasses
         self.log.debug("Deleting directory: path=%s", path)
 
-        info = self.info(path)
-        if info["type"] != "directory":
+        if not self.isdir(path):
+            # Raises FileNotFoundError if the path is missing
+            self.info(path)
             raise not_a_directory_error(path)
 
     def open(

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/src/fsspec_databricks/dbfs.py

@@ -8,7 +8,7 @@ from databricks.sdk.config import Config
 from databricks.sdk.service.files import FileInfo
 
 from .base import AbstractDatabricksFileSystem
-from .error import error_mapping, file_exists_error, is_a_directory_error
+from .error import error_mapping, is_a_directory_error
 from .path import parse_dbfs_path
 from .utils import to_datetime
 
@@ -80,15 +80,8 @@ class DBFS(AbstractDatabricksFileSystem):
 
         source_info = self.info(path1)
         if source_info["type"] == "directory":
-            try:
-                dest_info = self.info(path2)
-                if dest_info["type"] == "directory":
-                    return
-                else:
-                    raise file_exists_error(path2)
-            except FileNotFoundError:
-                self.mkdirs(path2)
-                return
+            self._cp_directory(path2)
+            return
 
         with error_mapping(path1):
             self.client.dbfs.copy(

{fsspec_databricks-0.2.0 → fsspec_databricks-0.3.0}/src/fsspec_databricks/spec.py

@@ -10,7 +10,7 @@ from fsspec.spec import AbstractFileSystem
 
 from .base import AbstractDatabricksFileSystem, root_dir_info
 from .dbfs import DBFS
-from .error import file_exists_error, file_not_found_error
+from .error import file_not_found_error
 from .local import DatabricksLocalFileSystem
 from .path import (
     FileSystemType,
@@ -293,23 +293,6 @@ class DatabricksFileSystem(AbstractDatabricksFileSystem):
             self.__local_fs = None
             super().close()
 
-    def __getstate__(self):
-        state = super().__getstate__()
-        del state["_DatabricksFileSystem__local_fs"]
-        del state["_DatabricksFileSystem__workspace_fs"]
-        del state["_DatabricksFileSystem__volume_fs"]
-        del state["_DatabricksFileSystem__legacy_fs"]
-        del state["_running_in_workspace"]
-        return state
-
-    def __setstate__(self, state):
-        super().__setstate__(state)
-        self.__workspace_fs = None
-        self.__volume_fs = None
-        self.__legacy_fs = None
-        self.__local_fs = None
-        self._running_in_workspace = None
-
     @property
     def _volume_fs(self) -> VolumeFileSystem:
         """Get or create the UnityCatalogVolumeFileSystem instance."""
@@ -458,6 +441,26 @@ class DatabricksFileSystem(AbstractDatabricksFileSystem):
         fs, p = self._resolve(path)
         return fs.exists(p, **kwargs)
 
+    def isdir(self, path) -> bool:
+        if self._strip_protocol(path) == "/":
+            return True
+
+        try:
+            fs, p = self._resolve(path)
+        except FileNotFoundError:
+            return False
+        return fs.isdir(p)
+
+    def isfile(self, path) -> bool:
+        if self._strip_protocol(path) == "/":
+            return False
+
+        try:
+            fs, p = self._resolve(path)
+        except FileNotFoundError:
+            return False
+        return fs.isfile(p)
+
     def created(self, path):
         fs, p = self._resolve(path)
         return fs.created(p)
@@ -475,15 +478,8 @@ class DatabricksFileSystem(AbstractDatabricksFileSystem):
         else:
             source_info = self.info(path1)
             if source_info["type"] == "directory":
-                try:
-                    dest_info = self.info(path2)
-                    if dest_info["type"] == "directory":
-                        return
-                    else:
-                        raise file_exists_error(path2)
-                except FileNotFoundError:
-                    self.mkdirs(path2)
-                    return
+                self._cp_directory(path2)
+                return
 
             src_fs, src_path = self._resolve(path1)
             dst_fs, dst_path = self._resolve(path2)