fsspec-databricks 0.2.0__tar.gz → 0.2.1__tar.gz

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/.gitignore

@@ -82,9 +82,6 @@ ENV/
 # vi*
 *.swp
 
-# UV
-uv.lock
-
 build/
 downstream/*
 

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/DEVELOPMENT.md

@@ -105,21 +105,60 @@ The workflows are in `.github/workflows/`:
 
 Each run's work directories are `<root>/<branch>/<Python version>` under the environment's roots.
 
+### Supply chain
+
+The test jobs hold the workspaces' credentials, so what they run is fixed:
+
+- `uv.lock` is committed, with every package's hash. CI installs with `uv sync --locked`, which fails if the lock is
+  out of date.
+- Actions are pinned to commit SHAs. setup-uv installs only uv versions whose checksums it knows (`latest-known`).
+- Dependabot updates `uv.lock` and the Actions' SHAs weekly.
+- Workflows run only on pushes and releases, never on pull requests from forks.
+- Each workflow's token can only read the repository. Publishing adds `id-token: write`, for PyPI's trusted
+  publishing.
+- The release build uses no cache.
+
+ruff, run with `uvx`, is the exception: CI runs its latest version, in a job without secrets.
+
 ### Environments
 
 Each GitHub environment is a Databricks workspace. Create at least `default`, and `azure`, `aws` or `gcp` as needed.
 The tests authenticate as a service principal with read-write access to the work directories.
 
-| Secret                     | Value                                         |
-|----------------------------|-----------------------------------------------|
-| `DATABRICKS_HOST`          | The workspace's URL                           |
-| `DATABRICKS_CLIENT_ID`     | The service principal's client ID             |
-| `DATABRICKS_CLIENT_SECRET` | The service principal's client secret         |
+Every environment has these:
+
+| Name                                               | Value                                          |
+|----------------------------------------------------|------------------------------------------------|
+| `DATABRICKS_HOST` (secret)                         | The workspace's URL                            |
+| `FSSPEC_DATABRICKS_VOLUME_TEST_ROOT` (variable)    | The POSIX path of the Volume work directory    |
+| `FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT` (variable) | The POSIX path of the Workspace work directory |
+
+The variable `DATABRICKS_AUTH_TYPE` chooses how the tests authenticate:
+
+| `DATABRICKS_AUTH_TYPE` | Authentication                                        | Also needs                                                       |
+|------------------------|-------------------------------------------------------|------------------------------------------------------------------|
+| Not set                | The service principal's client secret                 | `DATABRICKS_CLIENT_ID` and `DATABRICKS_CLIENT_SECRET` (secrets)  |
+| `github-oidc`          | Databricks federation policy                          | `DATABRICKS_CLIENT_ID` (secret), `DATABRICKS_TOKEN_AUDIENCE`     |
+| `github-oidc-azure`    | Federated credential of a Microsoft Entra ID app      | `ARM_CLIENT_ID`, the app's ID                                    |
+
+Prefer federation: it needs no stored secret. GitHub's OIDC token identifies each environment by the subject
+`repo:sugibuchi/fsspec-databricks:environment:<name>`. The federation's trust must name that subject and the issuer
+`https://token.actions.githubusercontent.com`.
 
-| Variable                                | Value                                         |
-|-----------------------------------------|-----------------------------------------------|
-| `FSSPEC_DATABRICKS_VOLUME_TEST_ROOT`    | The POSIX path of the Volume work directory    |
-| `FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT` | The POSIX path of the Workspace work directory |
+- **`github-oidc`**: a federation policy on the service principal, in the Databricks account console. Its audience is
+  `DATABRICKS_TOKEN_AUDIENCE`, e.g. the account ID.
+- **`github-oidc-azure`**: a federated credential on the Entra ID app, with the audience `api://AzureADTokenExchange`.
+  The app is the workspace's service principal. This needs no access to the account console.
+
+With federation, the client secret is not passed to the tests, even if the environment still has one.
+
+The environments are:
+
+| Environment | Workspace                        | Authentication      |
+|-------------|----------------------------------|---------------------|
+| `default`   | Databricks Free Edition          | Client secret       |
+| `azure`     | Azure Databricks                 | `github-oidc-azure` |
+| `gcp`       | Databricks on Google Cloud       | `github-oidc`       |
 
 At the repository level:
 
@@ -138,6 +177,12 @@ The version comes from the git tag, through hatch-vcs.
 
 `uv publish` authenticates to PyPI with the workflow's OIDC token, through trusted publishing.
 
+Repository rules protect the release path:
+
+- The ruleset "Release tags" lets only repository admins create, move or delete `v*` tags.
+- The `pypi` environment deploys only from `v*` tags.
+- The ruleset "Default branch" blocks force-pushes to `main` and its deletion.
+
 ## Design
 
 ### One file system for all of Databricks' paths
@@ -178,7 +223,8 @@ A `WorkspaceClient` given as `client` is kept, and is not picklable. Other ways
 
 ### Known limits
 
-- **Multipart upload uses an undocumented API.** Databricks may change it without notice.
+- **Multipart upload is not in the REST reference.** The Databricks SDK for Python uses the same endpoints, as an
+  experimental feature, in `databricks/sdk/mixins/files.py`. Compare with it when the API changes.
 - **Workspace files are at most 10 MB.** The Workspace API's limit.
 - **No append or update modes on Volumes.** `ab` and `r+b` are refused.
 - **No listing cache.** Each call reaches the workspace.
@@ -273,10 +319,22 @@ The APIs called depend on the depth of the path:
 | `/Volumes/<catalog>`                   | `catalogs.get`                                            | `schemas.list`  |
 | `/Volumes/<catalog>/<schema>`          | `schemas.get`                                             | `volumes.list`  |
 | `/Volumes/<catalog>/<schema>/<volume>` | `volumes.read`                                            | `dbfs.list`     |
-| Deeper                                 | `files.get_directory_metadata`, else `files.get_metadata` | `dbfs.list`     |
+| Deeper                                 | `files.get_metadata`, else `files.get_directory_metadata` | `dbfs.list`     |
 
 `include_browse` is passed to the Unity Catalog APIs.
 
+Each metadata call is a round trip, so the checks inside a volume make as few as they can:
+
+| Check                                         | Calls                                           |
+|-----------------------------------------------|-------------------------------------------------|
+| `info` of a file, so also `size` and reading  | 1: `files.get_metadata`                         |
+| `info` of a directory                         | 2: `files.get_metadata`, then the directory's   |
+| `isdir`, so also `mkdir` and `rmdir`          | 1: `files.get_directory_metadata`               |
+| `isfile`                                      | 1: `files.get_metadata`                         |
+| Opening for writing                           | 1: `isdir`, to refuse directories               |
+
+`DatabricksFileSystem` passes `isdir` and `isfile` to the backends. `test_volume_fs_metadata_calls` counts the calls.
+
 #### The IO event loop
 
 Each `VolumeFileSystem` runs an asyncio event loop in a daemon thread, `dbfs-io-thread`, started on first use. It

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: fsspec-databricks
-Version: 0.2.0
+Version: 0.2.1
 Summary: `fsspec`-based file system interface for Databricks file system"s"
 Project-URL: Homepage, https://github.com/sugibuchi/fsspec-databricks
 Author-email: Tsuyoshi SUGIBUCHI <sugibuchi@gmail.com>
@@ -117,7 +117,8 @@ to Workspace files only in workspaces with legacy DBFS disabled. See
 
 **Early beta.** The API and behaviour may change.
 
-* Writing Volume files uses Databricks' multipart upload API, which is undocumented and may change without notice.
+* Writing Volume files uses the Files API's multipart upload. The Databricks REST reference does not document it, but
+  the official Databricks SDK for Python uses it, as an experimental feature.
 * Databricks on AWS is not tested.
 * Legacy DBFS is not tested.
 

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/README.md

@@ -92,7 +92,8 @@ to Workspace files only in workspaces with legacy DBFS disabled. See
 
 **Early beta.** The API and behaviour may change.
 
-* Writing Volume files uses Databricks' multipart upload API, which is undocumented and may change without notice.
+* Writing Volume files uses the Files API's multipart upload. The Databricks REST reference does not document it, but
+  the official Databricks SDK for Python uses it, as an experimental feature.
 * Databricks on AWS is not tested.
 * Legacy DBFS is not tested.
 

fsspec_databricks-0.2.1/SECURITY.md

@@ -0,0 +1,30 @@
+# Security policy
+
+## Supported versions
+
+Security fixes are made in the latest release only.
+
+## Reporting a vulnerability
+
+Report vulnerabilities privately, through
+[GitHub's private vulnerability reporting](https://github.com/sugibuchi/fsspec-databricks/security/advisories/new).
+Do not open a public issue.
+
+Include:
+
+- the version of `fsspec-databricks`, and of `databricks-sdk`;
+- the steps to reproduce, or a proof of concept;
+- the impact, e.g. leaked credentials or access to files beyond the user's permissions.
+
+Never include real credentials, tokens or workspace URLs in a report.
+
+## Scope
+
+`fsspec-databricks` accesses Databricks with the credentials of its user, through the Databricks SDK. In scope are,
+for example:
+
+- credentials or tokens written to logs, exceptions or pickled file systems;
+- access to files or paths beyond what the given path names;
+- requests sent to hosts other than the workspace, its storage proxy, or the URLs Databricks returns.
+
+Vulnerabilities of Databricks itself or of the Databricks SDK go to Databricks.

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/pyproject.toml

@@ -73,6 +73,15 @@ source = [
 source = "vcs"
 raw-options = { 'version_scheme' = 'post-release' }
 
+[tool.hatch.build.targets.sdist]
+# Files for development only
+exclude = [
+    ".github",
+    ".pre-commit-config.yaml",
+    "CLAUDE.md",
+    "uv.lock",
+]
+
 [tool.ruff]
 exclude = [
     ".tox",

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/src/fsspec_databricks/base.py

@@ -213,15 +213,17 @@ class AbstractDatabricksFileSystem(AbstractFileSystem):
             "Making directory: path=%s, create_parents=%s", path, create_parents
         )
 
-        try:
-            info = self.info(self._parent(path))
-            if info["type"] != "directory":
-                raise not_a_directory_error(self._parent(path))
-        except FileNotFoundError:
-            if create_parents:
-                return
-            else:
-                raise
+        parent = self._parent(path)
+        if not self.isdir(parent):
+            # Tell a missing parent from a file
+            try:
+                self.info(parent)
+            except FileNotFoundError:
+                if create_parents:
+                    return
+                else:
+                    raise
+            raise not_a_directory_error(parent)
 
         if self.exists(path):
             raise file_exists_error(path)
@@ -248,8 +250,9 @@ class AbstractDatabricksFileSystem(AbstractFileSystem):
         # Actual operations must be implemented by subclasses
         self.log.debug("Deleting directory: path=%s", path)
 
-        info = self.info(path)
-        if info["type"] != "directory":
+        if not self.isdir(path):
+            # Raises FileNotFoundError if the path is missing
+            self.info(path)
             raise not_a_directory_error(path)
 
     def open(

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/src/fsspec_databricks/spec.py

@@ -458,6 +458,26 @@ class DatabricksFileSystem(AbstractDatabricksFileSystem):
         fs, p = self._resolve(path)
         return fs.exists(p, **kwargs)
 
+    def isdir(self, path) -> bool:
+        if self._strip_protocol(path) == "/":
+            return True
+
+        try:
+            fs, p = self._resolve(path)
+        except FileNotFoundError:
+            return False
+        return fs.isdir(p)
+
+    def isfile(self, path) -> bool:
+        if self._strip_protocol(path) == "/":
+            return False
+
+        try:
+            fs, p = self._resolve(path)
+        except FileNotFoundError:
+            return False
+        return fs.isfile(p)
+
     def created(self, path):
         fs, p = self._resolve(path)
         return fs.created(p)

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/src/fsspec_databricks/volume.py

@@ -20,7 +20,7 @@ from aiohttp import (
 from databricks.sdk import WorkspaceClient
 from databricks.sdk.client_types import HostType
 from databricks.sdk.config import Config
-from databricks.sdk.errors import BadRequest, DatabricksError
+from databricks.sdk.errors import BadRequest, NotFound
 from databricks.sdk.service.catalog import CatalogInfo, SchemaInfo, VolumeInfo
 from databricks.sdk.service.files import DirectoryEntry, GetMetadataResponse
 
@@ -403,19 +403,61 @@ class VolumeFileSystem(DBFS):
                     detail=True,
                 )
             else:
+                # Files first: they are asked about more often than directories
                 try:
-                    self.client.files.get_directory_metadata(full_path)
-                    return {
-                        "name": full_path.rstrip("/"),
-                        "size": None,
-                        "type": "directory",
-                        "created": None,
-                        "modified": None,
-                        "islink": False,
-                    }
-                except DatabricksError:
-                    metadata = self.client.files.get_metadata(full_path)
-                    return _map_info(metadata, detail=True, path=full_path)
+                    return self._file_info(full_path)
+                except NotFound:
+                    return self._directory_info(full_path)
+
+    def _file_info(self, full_path: str) -> dict[str, Any]:
+        """Info of a file in a volume, with one API call. Raises `NotFound` for other paths."""
+        metadata = self.client.files.get_metadata(full_path)
+        return _map_info(metadata, detail=True, path=full_path)
+
+    def _directory_info(self, full_path: str) -> dict[str, Any]:
+        """Info of a directory in a volume, with one API call. Raises `NotFound` for other paths."""
+        self.client.files.get_directory_metadata(full_path)
+        return {
+            "name": full_path.rstrip("/"),
+            "size": None,
+            "type": "directory",
+            "created": None,
+            "modified": None,
+            "islink": False,
+        }
+
+    def isdir(self, path):
+        try:
+            _, _, _, path_in_volume, full_path = parse_volume_path(path)
+        except ValueError:
+            return super().isdir(path)
+
+        if path_in_volume is None or path_in_volume == "/":
+            # The volumes root, catalogs, schemas and volumes
+            return super().isdir(path)
+
+        try:
+            with error_mapping(path):
+                self._directory_info(full_path)
+                return True
+        except OSError:
+            return False
+
+    def isfile(self, path):
+        try:
+            _, _, _, path_in_volume, full_path = parse_volume_path(path)
+        except ValueError:
+            return False
+
+        if path_in_volume is None or path_in_volume == "/":
+            return False
+
+        try:
+            with error_mapping(path):
+                self._file_info(full_path)
+                return True
+        except Exception:
+            return False
 
     def ls(self, path, detail=True, include_browse: bool | None = None, **kwargs):
         self.log.debug(
@@ -472,6 +514,28 @@ class VolumeFileSystem(DBFS):
                 else:
                     raise
 
+    def _check_open_target(self, path: str, mode: str) -> dict[str, Any] | None:
+        """Check that the path can be opened in the mode, and return its info if needed."""
+        if "r" not in mode and "x" not in mode:
+            # Writing only needs to refuse directories
+            if self.isdir(path):
+                raise is_a_directory_error(path)
+            return None
+
+        try:
+            info = self.info(path)
+        except FileNotFoundError:
+            info = None
+
+        if info is not None and info["type"] == "directory":
+            raise is_a_directory_error(path)
+        if "r" in mode and info is None:
+            raise file_not_found_error(path)
+        if "x" in mode and info is not None:
+            raise file_exists_error(path)
+
+        return info
+
     def _open(
         self,
         path,
@@ -487,18 +551,7 @@ class VolumeFileSystem(DBFS):
         if "a" in mode or "+" in mode:
             raise UnsupportedOperation("Append and read/write modes are not supported.")
 
-        try:
-            info = self.info(path)
-            if info["type"] == "directory":
-                raise is_a_directory_error(path)
-        except FileNotFoundError:
-            info = None
-
-        if "r" in mode and info is None:
-            raise file_not_found_error(path)
-
-        if "x" in mode and info is not None:
-            raise file_exists_error(path)
+        info = self._check_open_target(path, mode)
 
         auth = _workspace_authenticator(self.config)
 

{fsspec_databricks-0.2.0 → fsspec_databricks-0.2.1}/tests/test_volume_fs.py

@@ -1,4 +1,5 @@
 import asyncio
+import contextlib
 import errno
 import inspect
 import logging
@@ -6,7 +7,7 @@ import pickle
 from datetime import UTC, datetime, timedelta
 from io import BytesIO
 from random import randbytes
-from unittest.mock import patch
+from unittest.mock import MagicMock, patch
 
 import pytest
 from databricks.sdk import WorkspaceClient
@@ -182,6 +183,78 @@ def test_volume_fs_session_in_event_loop():
         assert session is fs._session
 
 
+class _CountingFilesAPI:
+    """A fake of `WorkspaceClient.files` for metadata, recording the calls."""
+
+    def __init__(self, files: list[str], directories: list[str]):
+        self.files = files
+        self.directories = directories
+        self.calls: list[tuple[str, str]] = []
+
+    def get_metadata(self, path):
+        self.calls.append(("file", path))
+        if path not in self.files:
+            raise NotFound(f"Not found: {path}")
+        return GetMetadataResponse(
+            content_length=123,
+            content_type="application/octet-stream",
+            last_modified="Wed, 21 Oct 2026 07:28:00 GMT",
+        )
+
+    def get_directory_metadata(self, path):
+        self.calls.append(("directory", path))
+        if path not in self.directories:
+            raise NotFound(f"Not found: {path}")
+
+
+def test_volume_fs_metadata_calls():
+    """Each check of a file or a directory makes as few API calls as possible."""
+    file = "/Volumes/c/s/v/dir/file.bin"
+    directory = "/Volumes/c/s/v/dir"
+    missing = "/Volumes/c/s/v/dir/missing"
+
+    files_api = _CountingFilesAPI(files=[file], directories=[directory])
+    client = MagicMock()
+    client.files = files_api
+
+    def calls_of(func):
+        files_api.calls.clear()
+        with contextlib.suppress(OSError):
+            func()
+        return list(files_api.calls)
+
+    with VolumeFileSystem(client=client) as fs:
+        # Files: one call
+        assert calls_of(lambda: fs.info(file)) == [("file", file)]
+        assert fs.info(file)["type"] == "file"
+        assert calls_of(lambda: fs.isfile(file)) == [("file", file)]
+        assert fs.isfile(file)
+        assert not fs.isfile(directory)
+
+        # Directories: one call to isdir, two to info
+        assert calls_of(lambda: fs.isdir(directory)) == [("directory", directory)]
+        assert fs.isdir(directory)
+        assert not fs.isdir(file)
+        assert calls_of(lambda: fs.info(directory)) == [
+            ("file", directory),
+            ("directory", directory),
+        ]
+        assert fs.info(directory)["type"] == "directory"
+
+        # Missing paths
+        with pytest.raises(FileNotFoundError):
+            fs.info(missing)
+
+        # Opening a directory for writing: one call
+        with pytest.raises(IsADirectoryError):
+            fs.open(directory, "wb")
+        assert calls_of(lambda: fs.open(directory, "wb")) == [("directory", directory)]
+
+        # Volumes, schemas and catalogs are directories, and never files
+        assert calls_of(lambda: fs.isfile("/Volumes/c/s/v")) == []
+        assert not fs.isfile("/Volumes/c/s/v")
+
+
 def test_volume_fs_workspace_id(client: WorkspaceClient):
     with VolumeFileSystem(client=client) as fs:
         workspace_id = fs.workspace_id

fsspec_databricks-0.2.0/.github/dependabot.yml

@@ -1,11 +0,0 @@
-version: 2
-updates:
-  - package-ecosystem: "pip"
-    directory: "/"
-    schedule:
-      interval: "weekly"
-
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "weekly"

fsspec_databricks-0.2.0/.github/workflows/check.yaml

@@ -1,33 +0,0 @@
-name: Check the source code
-
-on:
-  workflow_call:
-    inputs:
-      lint:
-        description: "Whether to run the linter."
-        type: boolean
-        required: false
-        default: true
-
-concurrency:
-  group: "check-${{ github.ref_name }}"
-  cancel-in-progress: true
-
-jobs:
-  lint:
-    name: Lint
-    if: ${{ inputs.lint }}
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v7
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@v7
-        with:
-          enable-cache: true
-
-      - name: Run ruff check
-        run: uvx ruff check .
-
-      - name: Run ruff format check
-        run: uvx ruff format --check .

fsspec_databricks-0.2.0/.github/workflows/on_push.yaml

@@ -1,85 +0,0 @@
-name: Actions for push events
-
-on:
-  push:
-    paths:
-      - ".github/**"
-      - "src/**"
-      - "tests/**"
-      - "pyproject.toml"
-
-concurrency:
-  group: "on-push-${{ github.ref_name }}"
-  cancel-in-progress: true
-
-jobs:
-  check:
-    name: Check the source code
-    uses: ./.github/workflows/check.yaml
-    with:
-      lint: true
-
-  test:
-    name: Test Python package
-    needs:
-      - check
-    uses: ./.github/workflows/test.yaml
-    with:
-      python-version: "3.14"
-      coverage: ${{ github.ref_name == 'main' }}
-      environment: "default"
-    secrets: inherit
-
-  cross_version_test:
-    name: Test Python package (Cross version)
-    if: ${{ github.ref_name == 'main' }}
-    needs:
-      - check
-    strategy:
-      matrix:
-        python-version:
-          - "3.11"
-          - "3.12"
-          - "3.13"
-    uses: ./.github/workflows/test.yaml
-    with:
-      python-version: ${{ matrix.python-version }}
-      coverage: false
-      environment: "default"
-    secrets: inherit
-
-  test_on_azure:
-    name: Test Python package on Azure Databricks
-    if: ${{ contains(vars.TEST_ENVIRONMENTS, 'azure') }}
-    needs:
-      - check
-    uses: ./.github/workflows/test.yaml
-    with:
-      python-version: "3.14"
-      coverage: ${{ github.ref_name == 'main' }}
-      environment: "azure"
-    secrets: inherit
-
-  test_on_aws:
-    name: Test Python package on Databricks on AWS
-    needs:
-      - check
-    if: ${{ contains(vars.TEST_ENVIRONMENTS, 'aws') }}
-    uses: ./.github/workflows/test.yaml
-    with:
-      python-version: "3.14"
-      coverage: ${{ github.ref_name == 'main' }}
-      environment: "aws"
-    secrets: inherit
-
-  test_on_gcp:
-    name: Test Python package on Databricks on GCP
-    needs:
-      - check
-    if: ${{ contains(vars.TEST_ENVIRONMENTS, 'gcp') }}
-    uses: ./.github/workflows/test.yaml
-    with:
-      python-version: "3.14"
-      coverage: ${{ github.ref_name == 'main' }}
-      environment: "gcp"
-    secrets: inherit

fsspec_databricks-0.2.0/.github/workflows/on_release.yaml

@@ -1,21 +0,0 @@
-name: Actions for release events
-
-on:
-  release:
-    types:
-      - published
-
-concurrency:
-  group: "on-release-${{ github.ref_name }}"
-  cancel-in-progress: true
-
-jobs:
-  publish:
-    name: Publish Python package
-    uses: ./.github/workflows/publish.yaml
-    with:
-      environment: "pypi"
-      python-version: "3.14"
-    permissions:
-      id-token: write
-      contents: read

fsspec_databricks-0.2.0/.github/workflows/publish.yaml

@@ -1,43 +0,0 @@
-name: Publish Python package
-
-on:
-  workflow_call:
-    inputs:
-      environment:
-        description: "The environment to publish to. Should be either 'pypi' or 'testpypi'."
-        type: string
-        required: true
-      python-version:
-        description: "The Python version to use for building and publishing the package."
-        type: string
-        required: true
-
-concurrency:
-  group: "publish-${{ github.ref_name }}"
-  cancel-in-progress: true
-
-jobs:
-  publish:
-    name: Publish Python package to PyPI
-    runs-on: ubuntu-latest
-    environment:
-      name: ${{ inputs.environment }}
-
-    steps:
-      - uses: actions/checkout@v7
-
-      - name: Set up Python
-        uses: actions/setup-python@v7
-        with:
-          python-version: ${{ inputs.python-version }}
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@v7
-        with:
-          enable-cache: true
-
-      - name: Build Python package
-        run: uv build
-
-      - name: Publish Python package
-        run: uv publish

fsspec_databricks-0.2.0/.github/workflows/test.yaml

@@ -1,139 +0,0 @@
-name: Test Python package
-run-name: Test the Python package
-
-on:
-  workflow_call:
-    inputs:
-      python-version:
-        description: "The Python version to use for testing the package."
-        type: string
-        required: true
-      coverage:
-        description: "Whether to run tests with coverage reporting."
-        type: boolean
-        required: false
-        default: false
-      environment:
-        description: "The environment to run tests"
-        type: string
-        required: true
-    secrets:
-      DATABRICKS_HOST:
-        required: true
-      DATABRICKS_CLIENT_ID:
-        required: true
-      DATABRICKS_CLIENT_SECRET:
-        required: true
-
-  workflow_dispatch:
-    inputs:
-      python-version:
-        description: "The Python version to use for testing the package."
-        type: string
-        required: true
-        default: "3.14"
-      coverage:
-        description: "Whether to run tests with coverage reporting."
-        type: boolean
-        required: true
-        default: false
-      environment:
-        description: "The environment to run tests"
-        type: choice
-        required: true
-        options:
-          - default
-          - azure
-          - aws
-          - gcp
-        default: default
-
-
-concurrency:
-  group: "test-${{ inputs.environment }}-${{ github.ref_name }}-${{ inputs.python-version }}"
-  cancel-in-progress: false
-
-jobs:
-
-  test:
-    name: Test Python package
-    runs-on: ubuntu-latest
-    environment: ${{ inputs.environment }}
-
-    steps:
-      - uses: actions/checkout@v7
-
-      - name: Set up Python ${{ inputs.python-version }}
-        uses: actions/setup-python@v7
-        with:
-          python-version: ${{ inputs.python-version }}
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@v7
-        with:
-          enable-cache: true
-
-      - name: Install the project
-        run: uv sync
-
-      - name: Set up env variables for working directories
-        shell: bash
-        run: |
-          test_path="${{ github.ref_name }}/${{ inputs.python-version }}"
-
-          has_error=no
-          if [[ "${{ vars.FSSPEC_DATABRICKS_VOLUME_TEST_ROOT }}" == "" ]]; then
-            echo "GitHub Actions variable FSSPEC_DATABRICKS_VOLUME_TEST_ROOT must be set."
-            has_error=yes
-          else
-            echo "FSSPEC_DATABRICKS_VOLUME_TEST_ROOT=${{ vars.FSSPEC_DATABRICKS_VOLUME_TEST_ROOT }}/${test_path}" >> $GITHUB_ENV
-          fi
-
-          if [[ "${{ vars.FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT }}" == "" ]]; then
-            echo "GitHub Actions variable FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT must be set."
-            has_error=yes
-          else
-            echo "FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT=${{ vars.FSSPEC_DATABRICKS_WORKSPACE_TEST_ROOT }}/${test_path}" >> $GITHUB_ENV
-          fi
-
-          if [[ "$has_error" = "yes" ]]; then
-            exit 1
-          fi
-
-      - name: Run test
-        if: ${{ !inputs.coverage }}
-        shell: bash
-        env:
-          DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
-          DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
-          DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
-        run: |
-          uv run pytest
-
-      - name: Run test with coverage
-        if: ${{ inputs.coverage }}
-        shell: bash
-        env:
-          DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
-          DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
-          DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
-        run: |
-          uv run pytest --cov \
-              --cov-report=term \
-              --cov-report=xml:coverage.xml \
-              --junitxml=junit.xml \
-              -o junit_family=legacy
-
-      - name: Upload coverage to Codecov
-        if: ${{ inputs.coverage && !cancelled() }}
-        uses: codecov/codecov-action@v7
-        with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          report_type: coverage
-
-      - name: Upload test results to Codecov
-        if: ${{ inputs.coverage && !cancelled() }}
-        uses: codecov/codecov-action@v7
-        with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          report_type: test_results

fsspec_databricks-0.2.0/.pre-commit-config.yaml

@@ -1,20 +0,0 @@
-exclude: >
-  (?x)^(
-      \.tox/.*
-  )$
-repos:
-
-  - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v6.0.0
-    hooks:
-      - id: trailing-whitespace
-      - id: end-of-file-fixer
-      - id: check-json
-      - id: check-yaml
-  - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.17.2
-    hooks:
-      - id: ruff-check
-        args: [ --fix, "--show-fixes"]
-      - id: ruff-format
-        types_or: [python]

fsspec_databricks-0.2.0/CLAUDE.md

@@ -1,108 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Commands
-
-All Python goes through uv: never call `python` or `pip` directly.
-
-```bash
-uv sync                                         # the package, editable, with the dev dependencies
-uv run pytest                                   # all tests: most need a Databricks workspace (see below)
-uv run pytest tests/test_path.py -k volume      # one file / matching tests
-uv run pytest --cov                             # with coverage of src/
-uvx ruff check .                                # lint (ruff is not a dev dependency; config in pyproject.toml)
-uvx ruff format .                               # format
-uvx pre-commit run --all-files                  # the hooks: ruff, whitespace, end of files, YAML
-```
-
-CI, in `.github/workflows/on_push.yaml`, runs on pushes:
-
-- ruff check and ruff format check, with the latest ruff;
-- pytest on Python 3.14 against the default workspace, with coverage on `main`;
-- on `main`, pytest on the other supported Python versions;
-- pytest on Azure, AWS or GCP workspaces named in the `TEST_ENVIRONMENTS` variable.
-
-Releases are published GitHub releases. DEVELOPMENT.md describes them.
-
-## Tests and layout
-
-DEVELOPMENT.md describes the layout, the tests and their setup. Read its Tests section before writing tests. The traps:
-
-- Most tests run against a real workspace. They need credentials and the two test root variables, from `.env` or the
-  environment.
-- The session fixtures `volume_test_root` and `workspace_test_root` delete everything in the test roots first. Never
-  point them at folders holding anything else.
-- Without credentials, run `tests/test_file.py`, `tests/test_volume.py`, `tests/test_path.py` and `tests/test_utils.py`.
-  `test_volume.py` and `test_file.py` use the fake Files API in `tests/dummy_api.py`.
-- A test of anything a workspace serves gets a check without one where possible, on the fake API or with mocks.
-- Test helpers shared by files are in `tests/utils.py`.
-- `tests/conftest.py` sets `cachable = False`, so each test gets new file system instances.
-
-## Architecture
-
-DEVELOPMENT.md's Architecture section describes each part: read it before changing one. The traps:
-
-- `DatabricksFileSystem` routes each call to a backend by path. It holds no file system logic of its own, except for
-  the root folder and transfers between backends.
-- Its methods get the backend and the path to pass with `_resolve(path)`. Inside a workspace, that path is a local
-  path, and names returned are mapped back with `from_local_path`.
-- Base class methods in `base.py` only check preconditions. Subclasses call `super()` first, then the API.
-- Wrap Databricks SDK calls in `error_mapping(path)`, so that callers get `OSError` subclasses.
-- Parse paths with the helpers of `path.py`. Every method takes `dbfs:/` URLs and POSIX paths alike.
-- aiohttp objects are created and used in the Volume backend's IO event loop. `TCPConnector` needs a running loop.
-- Never block on the IO loop from code running in it. File objects' synchronous methods raise there.
-- `DatabricksFileSystem.__init__` takes every parameter of the SDK's `WorkspaceClient`, in its order.
-  `test_dbx_fs_accepts_workspace_client_params` fails when the SDK adds one: add it to the signature, the docstring
-  and `config_params`.
-- File systems must stay picklable. A new attribute holding a client, a loop, a thread or a session is dropped in
-  `__getstate__` and reset in `__setstate__`.
-- Multipart upload uses an undocumented Databricks API. Keep its fallback to resumable upload working.
-- Volume write block sizes are multiples of 256 KB.
-- Workspace files are cached whole in memory, up to the API's 10 MB.
-- Chatty debug logs go under `if self.verbose_debug_log:`.
-
-## Conventions
-
-### Code
-
-- Comments are short, and say why the code does something not obvious. Keep that density and tone.
-- Type hints use built-in generics and `X | None`.
-- Each module ends with `__all__`.
-
-### Docstrings
-
-Docstrings of the public API are in numpy style: `Parameters`, `Returns`, `Raises`. The public API is what users
-construct or call:
-
-- `DatabricksFileSystem` and `use`;
-- `VolumeFileSystem`, `WorkspaceFileSystem` and `DBFS`, with their constructors and `open`.
-
-Internal docstrings stay one line or short prose: file objects, helpers and underscore names.
-
-### Writing docs
-
-Write docs so that people can review them. That covers this file, DEVELOPMENT.md and the user docs.
-
-- One idea per sentence: about 30 words, never more than 50.
-- At most one parenthesis per sentence, never nested.
-- Three or more items make a list.
-- Add a new fact as its own sentence or bullet where it belongs. Never append it to an existing sentence.
-- Name an identifier only when the reader needs it to find the code. A function's details belong in its docstring.
-- Describe what is, not what was: git keeps the history.
-- When editing a passage that breaks these rules, rewrite it.
-
-### Where docs go
-
-- README.md is the quick start: features, installation, paths, compatibility, status and limitations.
-- USAGE.md has the user-facing details: operations, each backend's behaviour, and the options.
-- AUTHENTICATION.md covers the ways to configure authentication.
-- DEVELOPMENT.md has setup, tests, CI, releasing, the design choices and the architecture.
-
-Update the docs when behaviour, parameters or options change. Keep user docs short and compact: what users do, not how
-it works or why.
-
-### Links
-
-README.md, USAGE.md and AUTHENTICATION.md are user docs. README.md uses absolute GitHub URLs for links to files, as
-PyPI shows it but cannot resolve relative paths. The other docs are read on GitHub only, so they use relative paths.