dlt-filesystem 0.18.0__tar.gz

dlt_filesystem-0.18.0/PKG-INFO

@@ -0,0 +1,114 @@
+Metadata-Version: 2.4
+Name: dlt-filesystem
+Version: 0.18.0
+Summary: Filesystem and blob storage sources and destinations for dlt, across 13 file formats.
+Maintainer-email: Andreas Motl <andreas.motl@panodata.org>
+License-Expression: MIT
+Project-URL: Issues, https://github.com/panodata/omniload/issues
+Project-URL: Repository, https://github.com/panodata/omniload
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: OS Independent
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: Unix
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Database
+Classifier: Topic :: Internet
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: System :: Archiving
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: adlfs
+Requires-Dist: azure-storage-blob
+Requires-Dist: dlt<1.31,>=1.22
+Requires-Dist: dropboxdrivefs
+Requires-Dist: fsspec<2026.10,>=2024.6
+Requires-Dist: fsspec-databricks>=0.1.10
+Requires-Dist: gcsfs
+Requires-Dist: gdrive-fsspec
+Requires-Dist: lakefs-spec<0.16
+Requires-Dist: msgraphfs
+Requires-Dist: ocifs
+Requires-Dist: ossfs
+Requires-Dist: paramiko
+Requires-Dist: pyarrow<26,>=18
+Requires-Dist: s3fs>=2024.6
+Requires-Dist: smbprotocol; platform_system == "Linux"
+Requires-Dist: smbprotocol[kerberos]; platform_system != "Linux"
+Requires-Dist: verlib2
+Requires-Dist: webdav4[fsspec]
+Provides-Extra: iterable
+Requires-Dist: cbor2<7; extra == "iterable"
+Requires-Dist: iterabledata<2,>=1.0.15; extra == "iterable"
+Requires-Dist: lxml<7; extra == "iterable"
+Requires-Dist: msgpack<2; extra == "iterable"
+Requires-Dist: pyyaml<7; extra == "iterable"
+Provides-Extra: vortex
+Requires-Dist: vortex-data<1,>=0.86; python_version >= "3.11" and extra == "vortex"
+
+# dlt-filesystem
+
+Filesystem and blob storage sources and destinations for
+[dlt](https://github.com/dlt-hub/dlt).
+
+`dlt` ships its own filesystem source, and this package is the delta over it:
+
+- **16 readers behind 19 routed format keys**, against dlt's four. CSV (pyarrow,
+  DuckDB and a headerless variant), JSON and JSONL, Parquet, ORC, Avro, Feather,
+  BSON, spreadsheets (`xlsx` and `ods`), XML, YAML, MessagePack and CBOR.
+- **A `filesystem` resource that refuses to load nothing.** A concrete selection
+  matching no file raises rather than returning an empty table; globs stay
+  empty-safe.
+- **A modification-date resolver that covers the schemes dlt's table does not**,
+  including `r2`, `oss`, `hdfs`, `smb`, `ftp` and `webdav`, and a pyarrow-backed
+  client addressed as `s3://`.
+- **An Arrow `readinto` shim**, without which reading a `.gz` fails wherever
+  `isal` is importable.
+
+The entry points are a superset of dlt's own, so a pipeline already on
+`dlt.sources.filesystem` can move across without changing its call.
+
+## Install
+
+```shell
+pip install dlt-filesystem
+```
+
+The long-tail formats (XML, YAML, MessagePack, CBOR) carry their decoders in an
+extra:
+
+```shell
+pip install 'dlt-filesystem[iterable]'
+```
+
+## Usage
+
+```python
+import dlt
+from dlt_filesystem.source.adapter import readers
+
+pipeline = dlt.pipeline(destination="duckdb", dataset_name="inbox")
+pipeline.run(
+    readers(bucket_url="s3://bucket/prefix", file_glob="*.parquet").read_parquet()
+)
+```
+
+## Documentation
+
+<https://omniload.readthedocs.io/supported-sources/filesystem.html>
+
+## License
+
+MIT. See [LICENSE](https://github.com/panodata/omniload/blob/main/LICENSE).

dlt_filesystem-0.18.0/README.md

@@ -0,0 +1,54 @@
+# dlt-filesystem
+
+Filesystem and blob storage sources and destinations for
+[dlt](https://github.com/dlt-hub/dlt).
+
+`dlt` ships its own filesystem source, and this package is the delta over it:
+
+- **16 readers behind 19 routed format keys**, against dlt's four. CSV (pyarrow,
+  DuckDB and a headerless variant), JSON and JSONL, Parquet, ORC, Avro, Feather,
+  BSON, spreadsheets (`xlsx` and `ods`), XML, YAML, MessagePack and CBOR.
+- **A `filesystem` resource that refuses to load nothing.** A concrete selection
+  matching no file raises rather than returning an empty table; globs stay
+  empty-safe.
+- **A modification-date resolver that covers the schemes dlt's table does not**,
+  including `r2`, `oss`, `hdfs`, `smb`, `ftp` and `webdav`, and a pyarrow-backed
+  client addressed as `s3://`.
+- **An Arrow `readinto` shim**, without which reading a `.gz` fails wherever
+  `isal` is importable.
+
+The entry points are a superset of dlt's own, so a pipeline already on
+`dlt.sources.filesystem` can move across without changing its call.
+
+## Install
+
+```shell
+pip install dlt-filesystem
+```
+
+The long-tail formats (XML, YAML, MessagePack, CBOR) carry their decoders in an
+extra:
+
+```shell
+pip install 'dlt-filesystem[iterable]'
+```
+
+## Usage
+
+```python
+import dlt
+from dlt_filesystem.source.adapter import readers
+
+pipeline = dlt.pipeline(destination="duckdb", dataset_name="inbox")
+pipeline.run(
+    readers(bucket_url="s3://bucket/prefix", file_glob="*.parquet").read_parquet()
+)
+```
+
+## Documentation
+
+<https://omniload.readthedocs.io/supported-sources/filesystem.html>
+
+## License
+
+MIT. See [LICENSE](https://github.com/panodata/omniload/blob/main/LICENSE).

dlt_filesystem-0.18.0/pyproject.toml

@@ -0,0 +1,113 @@
+[build-system]
+build-backend = "setuptools.build_meta"
+requires = [
+  "setuptools>=77",
+  "versioningit",
+]
+
+[project]
+name = "dlt-filesystem"
+description = "Filesystem and blob storage sources and destinations for dlt, across 13 file formats."
+readme = "README.md"
+license = "MIT"
+maintainers = [
+  { name = "Andreas Motl", email = "andreas.motl@panodata.org" },
+]
+requires-python = ">=3.10"
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Information Technology",
+  "Intended Audience :: Science/Research",
+  "Intended Audience :: System Administrators",
+  "Operating System :: MacOS :: MacOS X",
+  "Operating System :: Microsoft :: Windows",
+  "Operating System :: OS Independent",
+  "Operating System :: POSIX :: Linux",
+  "Operating System :: Unix",
+  "Programming Language :: Python",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Programming Language :: Python :: 3.14",
+  "Topic :: Database",
+  "Topic :: Internet",
+  "Topic :: Software Development :: Libraries",
+  "Topic :: System :: Archiving",
+  "Topic :: Utilities",
+]
+dynamic = [ "version" ]
+# `dlt` plus what reading and writing files over fsspec costs. Everything a single
+# format or transport needs beyond this is imported inside the function that needs
+# it, so an install carries the transports and pays for a format only on use.
+dependencies = [
+  "adlfs",
+  "azure-storage-blob",
+  "dlt>=1.22,<1.31",
+  "dropboxdrivefs",
+  "fsspec>=2024.6,<2026.10",
+  "fsspec-databricks>=0.1.10",
+  "gcsfs",
+  "gdrive-fsspec",
+  "lakefs-spec<0.16",
+  "msgraphfs",
+  "ocifs",
+  "ossfs",
+  "paramiko",
+  "pyarrow>=18,<26",
+  "s3fs>=2024.6",
+  "smbprotocol; platform_system=='Linux'",
+  "smbprotocol[kerberos]; platform_system!='Linux'",
+  "verlib2",
+  "webdav4[fsspec]",
+]
+# Long-tail file formats (MessagePack via iterabledata; CBOR via cbor2, XML via lxml, YAML via
+# PyYAML directly). iterabledata core is light (chardet, tqdm); each format's decoder is pulled
+# explicitly here (msgpack, cbor2, lxml, pyyaml). lxml and pyyaml are commonly present
+# transitively but are declared so the extra is self-contained. Note: iterabledata's import
+# package name is the generic `iterable`, which can shadow a same-named local package.
+# See dlt_filesystem/source/format/iterable_codec.py.
+optional-dependencies.iterable = [
+  "cbor2<7",
+  "iterabledata>=1.0.15,<2",
+  "lxml<7",
+  "msgpack<2",
+  "pyyaml<7",
+]
+# Vortex columnar files. vortex-data is a ~119 MB install that publishes wheels for Python
+# 3.11+ only, so it is opt-in rather than carried by every install of this package.
+optional-dependencies.vortex = [
+  "vortex-data>=0.86,<1; python_version>='3.11'",
+]
+urls.Issues = "https://github.com/panodata/omniload/issues"
+urls.Repository = "https://github.com/panodata/omniload"
+
+# The package carries no `__init__.py` at any level, so discovery has to be told that
+# a directory without one is still a package. Without this, setuptools finds nothing
+# and builds an empty wheel that `twine check` reports as PASSED.
+[tool.setuptools]
+packages.find.where = [ "src" ]
+packages.find.namespaces = true
+package-dir = { "" = "src" }
+
+[tool.pytest]
+ini_options.minversion = "2.0"
+ini_options.testpaths = [
+  "tests",
+]
+ini_options.addopts = """
+  -rfEXsS --strict-markers --verbosity=3
+  """
+# Declared here as well as in the consumer's configuration, because a run rooted at
+# this project reads this file alone and `--strict-markers` fails on an unknown one.
+ini_options.markers = [
+  "integration: requires Docker/testcontainers or external credentials",
+]
+ini_options.xfail_strict = true
+
+[tool.versioningit]
+vcs.method = "git-archive"
+vcs.default-tag = "v0.0.0"
+vcs.describe-subst = "$Format:%(describe:tags,match=v*)$"

dlt_filesystem-0.18.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

dlt_filesystem-0.18.0/src/dlt_filesystem/error.py

@@ -0,0 +1,14 @@
+from __future__ import annotations
+
+
+class MissingConnectorOption(Exception):
+    def __init__(self, option, connector):
+        super().__init__(f"{option} is required to connect to {connector}")
+
+
+class InvalidBlobTableError(Exception):
+    def __init__(self, source):
+        super().__init__(
+            f"Invalid source table for: {source}. "
+            "Ensure that the table is in the format {bucket-name}/{file glob}"
+        )

dlt_filesystem-0.18.0/src/dlt_filesystem/source/adapter.py

@@ -0,0 +1,180 @@
+# Copyright 2022-2025 ScaleVector
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Reads files in s3, gs or azure buckets using fsspec and provides convenience resources for chunked reading of various file formats"""
+
+from typing import Any, Dict, Iterable, Iterator, List, Optional, Tuple, Union
+
+import dlt
+from dlt.sources import DltResource
+from dlt.sources.credentials import FileSystemCredentials
+from dlt.sources.filesystem import FileItem, FileItemDict, fsspec_filesystem
+from fsspec import AbstractFileSystem
+
+from dlt_filesystem.source.error import NoFilesFoundError
+from dlt_filesystem.source.format import readers as reader_functions
+from dlt_filesystem.source.format.readers import ReadersSource
+from dlt_filesystem.source.format.registry import (
+    READER_REGISTRATIONS,
+    ReaderRegistration,
+)
+from dlt_filesystem.source.lister import glob_files
+
+from .model import FilesystemConfigurationResource
+
+
+def _resolve_reader(registration: ReaderRegistration):
+    """Resolve a registered reader name without making the registry import reader code."""
+    reader = getattr(reader_functions, registration.reader_name, None)
+    if not callable(reader):
+        raise ValueError(
+            f"Reader function {registration.reader_name!r} is not defined in "
+            "dlt_filesystem.source.format.readers"
+        )
+    return reader
+
+
+@dlt.source(_impl_cls=ReadersSource, spec=FilesystemConfigurationResource)
+def readers(
+    bucket_url: str = dlt.secrets.value,
+    credentials: Union[FileSystemCredentials, AbstractFileSystem] = dlt.secrets.value,
+    file_glob: Optional[str] = "*",
+    *,
+    kwargs: Optional[Dict[str, Any]] = None,
+    client_kwargs: Optional[Dict[str, Any]] = None,
+    incremental: Optional[dlt.sources.incremental[Any]] = None,
+) -> Tuple[DltResource, ...]:
+    """This source provides a few resources that are chunked file readers. Readers can be further parametrized before use
+       read_csv(chunksize, **pandas_kwargs)
+       read_json(chunksize)
+       read_jsonl(chunksize)
+       read_parquet(chunksize)
+
+    Args:
+        bucket_url (str): The url to the bucket.
+        credentials (FileSystemCredentials | AbstractFilesystem): The credentials to the filesystem of fsspec `AbstractFilesystem` instance.
+        file_glob (str, optional): The filter to apply to the files in glob format. by default lists all files in bucket_url non-recursively
+        kwargs (Optional[Dict[str, Any]]): Additional arguments passed to the fsspec constructor, ie. dict(use_ssl=True) for s3fs
+        client_kwargs (Optional[Dict[str, Any]]): Additional arguments passed to the underlying fsspec native client, ie. dict(verify="public.crt") for botocore
+        incremental (Optional[dlt.sources.incremental[Any]]): Defines an incremental cursor on the listed files, with `modification_date`
+            being the most common choice, which returns only files created since the previous run.
+    """
+    filesystem_resource = filesystem(
+        bucket_url,
+        credentials,
+        file_glob=file_glob,
+        kwargs=kwargs,
+        client_kwargs=client_kwargs,
+        incremental=incremental,
+    )
+
+    return tuple(
+        filesystem_resource
+        | dlt.transformer(
+            name=registration.reader_name,
+            max_table_nesting=registration.max_table_nesting,
+        )(_resolve_reader(registration))
+        for registration in READER_REGISTRATIONS
+    )
+
+
+@dlt.resource(
+    primary_key="file_url", spec=FilesystemConfigurationResource, standalone=True
+)
+def filesystem(
+    bucket_url: str = dlt.secrets.value,
+    credentials: Union[FileSystemCredentials, AbstractFileSystem] = dlt.secrets.value,
+    file_glob: Optional[str] = "*",
+    files_per_page: int = 100,
+    extract_content: bool = False,
+    require_file_match: bool = False,
+    filesystem_incremental: bool = False,
+    *,
+    kwargs: Optional[Dict[str, Any]] = None,
+    client_kwargs: Optional[Dict[str, Any]] = None,
+    incremental: Optional[dlt.sources.incremental[Any]] = None,
+) -> Iterator[List[FileItem]]:
+    """This resource lists files in `bucket_url` using `file_glob` pattern. The files are yielded as FileItem which also
+    provide methods to open and read file data. It should be combined with transformers that further process (ie. load files)
+
+    Args:
+        bucket_url (str): The url to the bucket.
+        credentials (FileSystemCredentials | AbstractFilesystem): The credentials to the filesystem of fsspec `AbstractFilesystem` instance.
+        file_glob (str, optional): The filter to apply to the files in glob format. by default lists all files in bucket_url non-recursively
+        files_per_page (int, optional): The number of files to process at once, defaults to 100.
+        extract_content (bool, optional): If true, the content of the file will be extracted if
+            false it will return a fsspec file, defaults to False.
+        require_file_match (bool, optional): Raise when the concrete source selection
+            matches no file. Defaults to False for direct uses of this resource.
+        filesystem_incremental (bool, optional): Resolve trustworthy modification
+            times when the listing itself does not carry one. Defaults to False.
+        kwargs (Optional[Dict[str, Any]]): Additional arguments passed to the fsspec constructor, ie. dict(use_ssl=True) for s3fs
+        client_kwargs (Optional[Dict[str, Any]]): Additional arguments passed to the underlying fsspec native client, ie. dict(verify="public.crt") for botocore
+        incremental (Optional[dlt.sources.incremental[Any]]): Defines an incremental cursor on the listed files, with `modification_date`
+            being the most common choice, which returns only files created since the previous run.
+            A cursor carrying `row_order` also orders the listing by its cursor field.
+
+    Returns:
+        Iterator[List[FileItem]]: The list of files.
+    """
+
+    fs_client: AbstractFileSystem
+    if isinstance(credentials, AbstractFileSystem):
+        # A caller who hands over a constructed filesystem has already spent
+        # `kwargs` and `client_kwargs` on building it, so both are ignored here,
+        # exactly as they are in dlt's own resource.
+        fs_client = credentials
+    else:
+        fs_client = fsspec_filesystem(
+            bucket_url, credentials, kwargs=kwargs, client_kwargs=client_kwargs
+        )[0]
+
+    file_models: Iterable[FileItem] = glob_files(
+        fs_client,
+        bucket_url,
+        file_glob or "**",
+        filesystem_incremental=filesystem_incremental,
+    )
+    if incremental and incremental.row_order:
+        # `row_order` is ascending or descending *in the direction `last_value_func`
+        # advances*, so it maps onto a raw sort only through that function: `max`
+        # advances upwards and `min` advances downwards, which inverts the
+        # comparison for `min`. Mirrors dlt's own expression.
+        reverse = (
+            incremental.row_order == "asc" and incremental.last_value_func is min
+        ) or (incremental.row_order == "desc" and incremental.last_value_func is max)
+        # The listing has to be materialised to be ordered. Only this branch pays
+        # for it; the default stays lazy.
+        file_models = sorted(
+            file_models,
+            key=lambda listed: listed[incremental.cursor_path],  # ty: ignore[invalid-key]
+            reverse=reverse,
+        )
+
+    matched_files = 0
+    files_chunk: List[FileItem] = []
+    for file_model in file_models:
+        matched_files += 1
+        file_dict = FileItemDict(file_model, fs_client)
+        if extract_content:
+            file_dict["file_content"] = file_dict.read_bytes()
+        files_chunk.append(file_dict)  # ty: ignore[invalid-argument-type]
+        # wait for the chunk to be full
+        if len(files_chunk) >= files_per_page:
+            yield files_chunk
+            files_chunk = []
+    if require_file_match and matched_files == 0:
+        raise NoFilesFoundError(bucket_url, file_glob or "**")
+    if files_chunk:
+        yield files_chunk

dlt_filesystem-0.18.0/src/dlt_filesystem/source/base.py

@@ -0,0 +1,89 @@
+from typing import Union
+from urllib.parse import urlparse
+
+
+class FilesystemSource:
+    """Shared capabilities for the filesystem-family sources.
+
+    Covers the local ``file://`` source and every remote transport
+    (``s3://``, ``gs://``, ``az://`` / ``adls://`` / ``abfss://``, ``sftp://``),
+    which all converge on the same reader after URI parsing.
+
+    Filesystem sources manage their own incremental behaviour
+    (``handles_incrementality`` is ``True``) and support opt-in file selection by
+    modification time (``supports_filesystem_incremental`` is ``True``). They
+    carry no resource-level write disposition, so a run-level disposition is safe
+    to apply: ``run_ingest`` honours an explicit ``--incremental-strategy append``
+    / ``replace`` for them (``honours_run_disposition`` is ``True``). Sources that
+    set their own resource-level disposition leave this ``False`` (the default)
+    so the run-level value never overrides theirs.
+    """
+
+    def handles_incrementality(self) -> bool:
+        return True
+
+    def honours_run_disposition(self) -> bool:
+        return True
+
+    def consumed_run_options(self) -> frozenset:
+        """Return the run options this source's ``dlt_source`` accepts by name.
+
+        Every other name in omniload's run vocabulary is filtered out before the
+        call, so it never reaches an fsspec or Arrow constructor as a stray
+        keyword. The two named here are resource options, not connector ones:
+        they configure how the reader resource is built (``FilesystemReference``)
+        and every ``dlt_source`` implementation in this family declares them
+        explicitly rather than reading them out of ``**kwargs``.
+        """
+        return frozenset({"filesystem_incremental", "column_types"})
+
+    def supports_filesystem_incremental(self) -> bool:
+        """Return whether the source supports file-level mtime selection."""
+        return True
+
+    def produces_multiple_tables(self, uri: str, table: str) -> bool:
+        """Return whether a workbook selection dispatches worksheet tables."""
+        from dlt_filesystem.source.error import UnsupportedEndpointError
+        from dlt_filesystem.source.format.readers import (
+            spreadsheet_selection_is_plural,
+        )
+        from dlt_filesystem.source.router import (
+            blob_hints,
+            determine_endpoint,
+            parse_uri,
+        )
+
+        parsed_uri = urlparse(uri)
+        _, path = parse_uri(parsed_uri, table)
+        try:
+            endpoint = determine_endpoint(table, path)
+        except (UnsupportedEndpointError, ValueError):
+            return False
+        return endpoint in {
+            "read_excel",
+            "read_ods",
+        } and spreadsheet_selection_is_plural(blob_hints(parsed_uri, table))
+
+    @staticmethod
+    def endpoint_namespace(endpoint: Union[str, None], default: str) -> str:
+        """
+        Return a normalized endpoint identity without credentials or query values.
+        It is used for incremental loading based on file modification times.
+
+        # TODO: Remove `default` argument again?
+        """
+        if not endpoint:
+            return default
+
+        parsed = urlparse(endpoint if "://" in endpoint else f"//{endpoint}")
+        host = parsed.hostname
+        if not host:
+            return default
+
+        host = host.lower()
+        if ":" in host:
+            host = f"[{host}]"
+        if parsed.port is not None:
+            host = f"{host}:{parsed.port}"
+
+        return f"{host}{parsed.path.rstrip('/')}"