dscanpy 0.1.0__tar.gz

dscanpy-0.1.0/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: dscanpy
+Version: 0.1.0
+Summary: Concurrent directory tree scanner for Python 3.12+
+License: MIT
+Requires-Python: >=3.12
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: System :: Filesystems
+Project-URL: Repository, https://github.com/yourusername/dscanpy
+Description-Content-Type: text/markdown
+
+# dscan
+
+`dscan` is a concurrent directory scanner for Python 3.12+. It wraps `os.scandir` in a thread pool with a work-stealing queue, exposing a filtering API that covers most of what you'd otherwise implement by hand on top of `os.walk`.
+
+Two modes: `scan_entries` yields raw `os.DirEntry` objects with minimal overhead; `scan` yields dataclass models with pre-computed metadata.
+
+---
+
+## Why concurrent scanning?
+
+On a local SSD, directory traversal is fast enough that threading adds more overhead than it saves. `scan_entries` still matches or edges out `os.walk`, but the real case for concurrency is **network-attached storage**.
+
+On SMB shares, NFS mounts, or any high-latency filesystem, each `scandir` call blocks waiting for a server response. `os.walk` does this serially — one directory at a time. dscan keeps multiple directories in-flight simultaneously, so workers aren't sitting idle while the network responds. On deep trees with many subdirectories, this compounds significantly.
+
+---
+
+## Benchmarks
+
+### Local SSD (~4M entries, MacBook)
+
+| | entries | time |
+|---|---|---|
+| `os.walk` (no stat) | 4,046,505 | 33.30s |
+| `os.walk` (+ stat) | 4,039,313 | 85.24s |
+| `dscan.scan_entries` | 4,046,502 | **31.90s** |
+| `dscan.scan` (models) | 4,014,758 | 140.15s |
+
+`scan_entries` is on par with bare `os.walk`. `scan` is slower because stat calls happen on the main thread serially — the workers parallelise `scandir`, not `stat`. Use `scan` when you want the structured output; use `scan_entries` when throughput matters.
+
+### Simulated network latency (5ms per directory)
+
+```python
+# rough simulation
+import time, os
+_real = os.scandir
+os.scandir = lambda p: (time.sleep(0.005), _real(p))[1]
+```
+
+| | time |
+|---|---|
+| `os.walk` | ~linear with directory count |
+| `dscan.scan_entries` | scales with `max_workers` |
+
+At 5ms latency per directory, a tree with 10,000 directories takes ~50s serially. With 16 workers dscan brings that to ~4s. The deeper and wider the tree, the bigger the difference.
+
+---
+
+## Installation
+
+```bash
+pip install dscan
+```
+
+Requires Python 3.12+. No other dependencies.
+
+---
+
+## Usage
+
+### Basic scan
+
+```python
+from dscan import scan
+
+for entry in scan("."):
+    print(f"{entry.name} - {entry.path}")
+```
+
+### Raw entries (lower overhead)
+
+```python
+from dscan import scan_entries
+
+for entry in scan_entries("~/Documents", max_depth=2):
+    if entry.is_file():
+        print(entry.name)
+```
+
+---
+
+## Filtering
+
+### Extensions
+
+```python
+# Only Python and Markdown files
+for file in scan(".", extensions={".py", ".md"}):
+    print(file.path)
+
+# Skip compiled files
+for file in scan(".", ignore_extensions={".bin", ".exe"}):
+    print(file.path)
+```
+
+### Glob patterns
+
+```python
+# Only test files
+for entry in scan(".", match="test_*"):
+    print(entry.name)
+
+# Skip hidden files and directories
+for entry in scan(".", ignore_pattern=".*"):
+    print(entry.name)
+```
+
+### Directory traversal
+
+```python
+# Immediate children only
+for entry in scan(".", max_depth=0):
+    print(entry.name)
+
+# Only descend into src/ and lib/
+for entry in scan(".", only_dirs=["src", "lib"]):
+    print(entry.path)
+
+# Skip specific directories
+# .git, .idea, .venv, __pycache__ are skipped by default
+for entry in scan(".", ignore_dirs=["node_modules", "dist"]):
+    print(entry.path)
+
+# Disable all default ignores
+for entry in scan(".", ignore_dirs=[]):
+    print(entry.path)
+```
+
+### Custom filter
+
+```python
+def is_large_file(entry):
+    return entry.is_file() and entry.stat().st_size > 1_000_000
+
+for entry in scan(".", custom_filter=is_large_file):
+    print(entry.name)
+```
+
+### Tuning workers
+
+```python
+# default is min(32, cpu_count * 2)
+# increase on high-latency mounts
+for entry in scan_entries("/mnt/nas", max_workers=32):
+    print(entry.path)
+```
+
+---
+
+## Data Models
+
+`scan()` returns `FileEntry` or `DirectoryEntry` dataclasses.
+
+### `FileEntry`
+
+| field | description |
+|---|---|
+| `name` | filename without extension |
+| `extension` | lowercase extension, no leading dot |
+| `path` | full path |
+| `dir_path` | containing directory |
+| `size` | bytes |
+| `created_at` | `datetime` |
+| `modified_at` | `datetime` |
+
+### `DirectoryEntry`
+
+| field | description |
+|---|---|
+| `name` | directory name |
+| `path` | full path |
+| `parent_path` | parent directory |
+| `created_at` | `datetime` |
+| `modified_at` | `datetime` |
+
+---
+
+## vs the stdlib
+
+| | `os.walk` | `pathlib.rglob` | `dscan` |
+|---|:---:|:---:|:---:|
+| Concurrent traversal | No | No | Yes |
+| Built-in models | No | No | Yes |
+| Depth limit | Manual | No | Yes |
+| Directory exclusions | Manual | No | Yes |
+
+---
+
+## License
+
+MIT

dscanpy-0.1.0/README.md

@@ -0,0 +1,190 @@
+# dscan
+
+`dscan` is a concurrent directory scanner for Python 3.12+. It wraps `os.scandir` in a thread pool with a work-stealing queue, exposing a filtering API that covers most of what you'd otherwise implement by hand on top of `os.walk`.
+
+Two modes: `scan_entries` yields raw `os.DirEntry` objects with minimal overhead; `scan` yields dataclass models with pre-computed metadata.
+
+---
+
+## Why concurrent scanning?
+
+On a local SSD, directory traversal is fast enough that threading adds more overhead than it saves. `scan_entries` still matches or edges out `os.walk`, but the real case for concurrency is **network-attached storage**.
+
+On SMB shares, NFS mounts, or any high-latency filesystem, each `scandir` call blocks waiting for a server response. `os.walk` does this serially — one directory at a time. dscan keeps multiple directories in-flight simultaneously, so workers aren't sitting idle while the network responds. On deep trees with many subdirectories, this compounds significantly.
+
+---
+
+## Benchmarks
+
+### Local SSD (~4M entries, MacBook)
+
+| | entries | time |
+|---|---|---|
+| `os.walk` (no stat) | 4,046,505 | 33.30s |
+| `os.walk` (+ stat) | 4,039,313 | 85.24s |
+| `dscan.scan_entries` | 4,046,502 | **31.90s** |
+| `dscan.scan` (models) | 4,014,758 | 140.15s |
+
+`scan_entries` is on par with bare `os.walk`. `scan` is slower because stat calls happen on the main thread serially — the workers parallelise `scandir`, not `stat`. Use `scan` when you want the structured output; use `scan_entries` when throughput matters.
+
+### Simulated network latency (5ms per directory)
+
+```python
+# rough simulation
+import time, os
+_real = os.scandir
+os.scandir = lambda p: (time.sleep(0.005), _real(p))[1]
+```
+
+| | time |
+|---|---|
+| `os.walk` | ~linear with directory count |
+| `dscan.scan_entries` | scales with `max_workers` |
+
+At 5ms latency per directory, a tree with 10,000 directories takes ~50s serially. With 16 workers dscan brings that to ~4s. The deeper and wider the tree, the bigger the difference.
+
+---
+
+## Installation
+
+```bash
+pip install dscan
+```
+
+Requires Python 3.12+. No other dependencies.
+
+---
+
+## Usage
+
+### Basic scan
+
+```python
+from dscan import scan
+
+for entry in scan("."):
+    print(f"{entry.name} - {entry.path}")
+```
+
+### Raw entries (lower overhead)
+
+```python
+from dscan import scan_entries
+
+for entry in scan_entries("~/Documents", max_depth=2):
+    if entry.is_file():
+        print(entry.name)
+```
+
+---
+
+## Filtering
+
+### Extensions
+
+```python
+# Only Python and Markdown files
+for file in scan(".", extensions={".py", ".md"}):
+    print(file.path)
+
+# Skip compiled files
+for file in scan(".", ignore_extensions={".bin", ".exe"}):
+    print(file.path)
+```
+
+### Glob patterns
+
+```python
+# Only test files
+for entry in scan(".", match="test_*"):
+    print(entry.name)
+
+# Skip hidden files and directories
+for entry in scan(".", ignore_pattern=".*"):
+    print(entry.name)
+```
+
+### Directory traversal
+
+```python
+# Immediate children only
+for entry in scan(".", max_depth=0):
+    print(entry.name)
+
+# Only descend into src/ and lib/
+for entry in scan(".", only_dirs=["src", "lib"]):
+    print(entry.path)
+
+# Skip specific directories
+# .git, .idea, .venv, __pycache__ are skipped by default
+for entry in scan(".", ignore_dirs=["node_modules", "dist"]):
+    print(entry.path)
+
+# Disable all default ignores
+for entry in scan(".", ignore_dirs=[]):
+    print(entry.path)
+```
+
+### Custom filter
+
+```python
+def is_large_file(entry):
+    return entry.is_file() and entry.stat().st_size > 1_000_000
+
+for entry in scan(".", custom_filter=is_large_file):
+    print(entry.name)
+```
+
+### Tuning workers
+
+```python
+# default is min(32, cpu_count * 2)
+# increase on high-latency mounts
+for entry in scan_entries("/mnt/nas", max_workers=32):
+    print(entry.path)
+```
+
+---
+
+## Data Models
+
+`scan()` returns `FileEntry` or `DirectoryEntry` dataclasses.
+
+### `FileEntry`
+
+| field | description |
+|---|---|
+| `name` | filename without extension |
+| `extension` | lowercase extension, no leading dot |
+| `path` | full path |
+| `dir_path` | containing directory |
+| `size` | bytes |
+| `created_at` | `datetime` |
+| `modified_at` | `datetime` |
+
+### `DirectoryEntry`
+
+| field | description |
+|---|---|
+| `name` | directory name |
+| `path` | full path |
+| `parent_path` | parent directory |
+| `created_at` | `datetime` |
+| `modified_at` | `datetime` |
+
+---
+
+## vs the stdlib
+
+| | `os.walk` | `pathlib.rglob` | `dscan` |
+|---|:---:|:---:|:---:|
+| Concurrent traversal | No | No | Yes |
+| Built-in models | No | No | Yes |
+| Depth limit | Manual | No | Yes |
+| Directory exclusions | Manual | No | Yes |
+
+---
+
+## License
+
+MIT

dscanpy-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "dscanpy"
+version = "0.1.0"
+description = "Concurrent directory tree scanner for Python 3.12+"
+readme = "README.md"
+requires-python = ">=3.12"
+license = {text = "MIT"}
+classifiers = [
+    "Programming Language :: Python :: 3.12",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Topic :: System :: Filesystems",
+]
+
+[project.urls]
+Repository = "https://github.com/yourusername/dscanpy"
+
+[tool.poetry]
+packages = [{include = "dscan", from = "src"}]
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[dependency-groups]
+dev = [
+    "ruff (>=0.15.5,<0.16.0)",
+    "ipykernel (>=7.2.0,<8.0.0)",
+    "jupyterlab (>=4.5.5,<5.0.0)"
+]

dscanpy-0.1.0/src/dscan/__init__.py

@@ -0,0 +1,5 @@
+from dscan.core import scan, scan_entries
+from dscan.models import FileEntry, DirectoryEntry
+from dscan.filter import FilterMode
+
+__all__ = ["scan", "scan_entries", "FileEntry", "DirectoryEntry", "FilterMode"]

dscanpy-0.1.0/src/dscan/core.py

@@ -0,0 +1,220 @@
+import logging
+import os
+from collections.abc import Callable, Iterator
+from pathlib import Path
+from dscan.models import FileEntry, DirectoryEntry
+from dscan.crawler import TreeCrawler
+from dscan.filter import FilterMode, ScanFilter
+
+logger = logging.getLogger(__name__)
+
+
+def scan_entries(
+    path: str | Path,
+    *,
+    # ── What to yield ────────────────────────────────────────────────────────
+    files_only: bool = False,
+    dirs_only: bool = False,
+    # ── Extension filtering (mutually exclusive) ──────────────────────────────
+    extensions: set[str] | list[str] | None = None,
+    ignore_extensions: set[str] | list[str] | None = None,
+    # ── Name-pattern filtering (mutually exclusive) ───────────────────────────
+    match: str | None = None,
+    ignore_pattern: str | None = None,
+    # ── Traversal ────────────────────────────────────────────────────────────
+    ignore_dirs: list[str] | None = None,
+    only_dirs: list[str] | None = None,
+    max_depth: int | None = None,
+    max_workers: int | None = None,
+    # ── Escape hatch ─────────────────────────────────────────────────────────
+    custom_filter: Callable[[os.DirEntry], bool] | None = None,
+) -> Iterator[os.DirEntry]:
+    """Scan a directory tree and yield matching entries.
+
+    Combines traversal and filtering into a single call. All keyword
+    arguments are optional — calling ``scan_entries(path)`` alone yields every
+    entry in the tree with sensible defaults.
+
+    Args:
+        path: Root directory to scan. Strings are accepted and resolved
+            automatically; non-existent or non-directory paths raise
+            ``ValueError``.
+
+        files_only: Yield only files (no directories).
+        dirs_only:  Yield only directories (no files).
+            ``files_only`` and ``dirs_only`` are mutually exclusive.
+
+        extensions: Allowlist of file extensions to yield, e.g.
+            ``{".py", ".md"}``. Leading dots are optional and matching
+            is case-insensitive. Has no effect on directory entries.
+        ignore_extensions: Denylist of extensions to suppress.
+            Mutually exclusive with ``extensions``.
+
+        match: Glob pattern — only yield entries whose name matches,
+            e.g. ``"test_*"``.
+        ignore_pattern: Glob pattern — suppress entries whose name
+            matches, e.g. ``".*"``.
+            Mutually exclusive with ``match``.
+
+        ignore_dirs: Directory names to skip when descending, e.g.
+            ``["node_modules", "dist"]``. Merged with the built-in
+            defaults (``.git``, ``.idea``, ``.venv``, ``__pycache__``).
+            Pass an empty list to disable all default ignores.
+        only_dirs: When provided, descend *only* into directories whose
+            name appears in this list. Mutually exclusive with
+            ``ignore_dirs``.
+        max_depth: How many levels deep to descend. ``0`` scans only the
+            root itself; ``None`` (default) is unlimited.
+        max_workers: Worker thread count. Defaults to
+            ``min(32, cpu_count * 2)``.
+
+        custom_filter: Optional callable ``(DirEntry) -> bool``. Return
+            ``False`` to drop an entry. Applied after all other filters.
+
+    Yields:
+        ``os.DirEntry`` for every entry that passes all active filters.
+
+    Raises:
+        ValueError: If ``path`` does not exist or is not a directory, or
+            if any pair of mutually-exclusive arguments are both supplied.
+
+    Examples:
+        Yield all Python files up to 3 levels deep::
+
+            for entry in scan_entries("~/projects", extensions={".py"}, max_depth=3):
+                print(entry.path)
+
+        Yield everything, skipping hidden files and ``dist`` folders::
+
+            for entry in scan_entries(
+                "/srv/app",
+                ignore_pattern=".*",
+                ignore_dirs=["dist"],
+            ):
+                print(entry.path)
+
+        Only descend into ``src`` and ``tests`` directories::
+
+            for entry in scan_entries(".", only_dirs=["src", "tests"], files_only=True):
+                print(entry.path)
+    """
+    # ── Validate mutually exclusive pairs ────────────────────────────────────
+    if files_only and dirs_only:
+        raise ValueError("files_only and dirs_only are mutually exclusive")
+    if extensions and ignore_extensions:
+        raise ValueError("extensions and ignore_extensions are mutually exclusive")
+    if match and ignore_pattern:
+        raise ValueError("match and ignore_pattern are mutually exclusive")
+    if ignore_dirs is not None and only_dirs is not None:
+        raise ValueError("ignore_dirs and only_dirs are mutually exclusive")
+
+    # ── Resolve path ─────────────────────────────────────────────────────────
+    root = _resolve_path(path)
+
+    # ── Build TreeCrawler ─────────────────────────────────────────────────────
+    if only_dirs is not None:
+        crawler = TreeCrawler(
+            dirs=only_dirs,
+            filter_mode=FilterMode.INCLUDE,
+            max_workers=max_workers,
+            max_depth=max_depth,
+        )
+    else:
+        # Merge caller-supplied ignores with the built-in defaults.
+        default_ignores = {".git", ".idea", ".venv", "__pycache__"}
+        extra = set(ignore_dirs) if ignore_dirs is not None else set()
+        merged = (default_ignores | extra) if ignore_dirs is None else extra
+        crawler = TreeCrawler(
+            dirs=list(merged),
+            filter_mode=FilterMode.IGNORE,
+            max_workers=max_workers,
+            max_depth=max_depth,
+        )
+
+    # ── Build ScanFilter ──────────────────────────────────────────────────────
+    scan_filter = ScanFilter(
+        only_files=files_only,
+        only_dirs=dirs_only,
+        extensions=_normalise_exts(extensions)
+        if extensions
+        else (_normalise_exts(ignore_extensions) if ignore_extensions else None),
+        extensions_mode=(
+            FilterMode.IGNORE if ignore_extensions else FilterMode.INCLUDE
+        ),
+        name_pattern=match or ignore_pattern,
+        name_pattern_mode=(FilterMode.IGNORE if ignore_pattern else FilterMode.INCLUDE),
+        custom=custom_filter,
+    )
+
+    return scan_filter.apply(crawler.scan(root))
+
+
+def scan(
+    path: str | Path,
+    *,
+    # ── What to yield ────────────────────────────────────────────────────────
+    files_only: bool = False,
+    dirs_only: bool = False,
+    # ── Extension filtering (mutually exclusive) ──────────────────────────────
+    extensions: set[str] | list[str] | None = None,
+    ignore_extensions: set[str] | list[str] | None = None,
+    # ── Name-pattern filtering (mutually exclusive) ───────────────────────────
+    match: str | None = None,
+    ignore_pattern: str | None = None,
+    # ── Traversal ────────────────────────────────────────────────────────────
+    ignore_dirs: list[str] | None = None,
+    only_dirs: list[str] | None = None,
+    max_depth: int | None = None,
+    max_workers: int | None = None,
+    # ── Escape hatch ─────────────────────────────────────────────────────────
+    custom_filter: Callable[[os.DirEntry], bool] | None = None,
+) -> Iterator[FileEntry | DirectoryEntry]:
+    """Like ``scan_entries``, but yields rich metadata models instead of raw
+    ``DirEntry`` objects.
+
+    See ``scan_entries`` for argument details and examples.
+
+    Yields:
+        FileEntry or DirectoryEntry, depending on the type of each entry.
+    """
+    for entry in scan_entries(
+        path,
+        files_only=files_only,
+        dirs_only=dirs_only,
+        extensions=extensions,
+        ignore_extensions=ignore_extensions,
+        match=match,
+        ignore_pattern=ignore_pattern,
+        ignore_dirs=ignore_dirs,
+        only_dirs=only_dirs,
+        max_depth=max_depth,
+        max_workers=max_workers,
+        custom_filter=custom_filter,
+    ):
+        if entry.is_file(follow_symlinks=False):
+            yield FileEntry.from_dir_entry(entry)
+        elif entry.is_dir(follow_symlinks=False):
+            yield DirectoryEntry.from_dir_entry(entry)
+
+
+# ── Internal helpers ──────────────────────────────────────────────────────────
+
+
+def _resolve_path(path: str | Path) -> Path:
+    """Resolve *path* to an absolute ``Path``, raising ``ValueError`` on failure."""
+    try:
+        p = Path(path).expanduser().resolve()
+    except Exception as e:
+        raise ValueError(f"Invalid path {path!r}: {e}") from e
+
+    if not p.exists():
+        raise ValueError(f"Path does not exist: {p}")
+    if not p.is_dir():
+        raise ValueError(f"Path is not a directory: {p}")
+
+    return p
+
+
+# ── Normalise extensions (ensure leading dot) ─────────────────────────────
+def _normalise_exts(exts: set[str] | list[str]) -> set[str]:
+    return {e if e.startswith(".") else f".{e}" for e in exts}

dscanpy-0.1.0/src/dscan/crawler.py

@@ -0,0 +1,275 @@
+import logging
+import os
+from pathlib import Path
+from queue import Queue, Empty
+from threading import Lock, Event
+from concurrent.futures import ThreadPoolExecutor
+from typing import Iterator, Literal
+from dscan.filter import FilterMode
+
+logger = logging.getLogger(__name__)
+
+
+class _ScanState:
+    """Thread-safe pending work counter for coordinating concurrent directory scanning.
+
+    Tracks the number of directories that are either queued or actively being
+    processed by a worker. Signals completion once all work is exhausted,
+    allowing the main thread to stop draining results.
+    """
+
+    def __init__(self) -> None:
+        self._lock = Lock()
+        self._pending = 0
+        self._done = Event()
+
+    def add(self, count: int) -> None:
+        """Register new units of work.
+
+        Must be called before putting new directories into the queue to avoid
+        a race condition where workers falsely signal completion.
+
+        Args:
+            count: Number of new directories to register as pending.
+        """
+        with self._lock:
+            self._pending += count
+
+    def complete(self) -> None:
+        """Mark one unit of work as finished.
+
+        Decrements the pending counter. If no work remains, sets the done
+        event so the main thread and all workers can exit cleanly.
+        """
+        with self._lock:
+            self._pending -= 1
+            if self._pending == 0:
+                logger.debug("All pending work exhausted — signalling done")
+                self._done.set()
+
+    @property
+    def is_done(self) -> bool:
+        """Returns True if all pending work has been completed."""
+        return self._done.is_set()
+
+
+class TreeCrawler:
+    """Crawls a directory tree concurrently using a shared work-stealing queue.
+
+    Workers pull directories from a shared queue. Newly discovered subdirectories
+    are pushed back into the queue for any idle worker to pick up, keeping all
+    threads busy regardless of tree depth or structure.
+
+    Example:
+        # Ignore specific directories (default mode):
+        crawler = TreeCrawler(dirs={".git", "node_modules"}, max_depth=3)
+
+        # Only descend into specific directories:
+        crawler = TreeCrawler(dirs={"src", "lib"}, filter_mode=FilterMode.INCLUDE)
+
+        for entry in crawler.scan(Path("/home/user")):
+            print(entry.path)
+
+    Attributes:
+        _dirs: Set of directory names used for filtering during traversal.
+        _filter_mode: Whether _dirs is treated as a block list or include list.
+        _max_workers: Number of worker threads used during scanning.
+        _max_depth: Maximum directory depth to scan. None means unlimited.
+    """
+
+    def __init__(
+        self,
+        dirs: list[str] | None = None,
+        filter_mode: FilterMode = FilterMode.IGNORE,
+        max_workers: int | None = None,
+        max_depth: int | None = None,
+    ) -> None:
+        """Initialises the TreeCrawler.
+
+        Args:
+            dirs: Directory names to filter during traversal. Behaviour depends
+                on filter_mode. Defaults to {".git", ".idea", ".venv",
+                "__pycache__"} when mode is FilterMode.IGNORE, or an empty set otherwise.
+            filter_mode: Controls how dirs is applied during descent:
+                - filter_mode.IGNORE  — skip directories whose name is in dirs.
+                - filter_mode.INCLUDE — only descend into directories whose name is in dirs.
+                Entries are still yielded regardless of this filter; it only
+                controls whether a directory is recursed into.
+            max_workers: Number of worker threads. Defaults to min(32, cpu_count * 2).
+            max_depth: Maximum depth to descend relative to root_path. Depth 0
+                scans only the root itself, depth 1 includes its immediate children,
+                and so on. None (default) means unlimited depth.
+        """
+        self._filter_mode = filter_mode
+        self._dirs = (
+            set(dirs)
+            if dirs is not None
+            else (
+                {".git", ".idea", ".venv", "__pycache__"}
+                if filter_mode == FilterMode.IGNORE
+                else set()
+            )
+        )
+        self._max_workers = max_workers or min(32, max(1, (os.cpu_count() or 4) * 2))
+        self._max_depth = max_depth
+        logger.debug(
+            f"TreeCrawler initialised — "
+            f"filter_mode={self._filter_mode!r}, "
+            f"dirs={self._dirs}, "
+            f"max_workers={self._max_workers}, "
+            f"max_depth={self._max_depth!r}"
+        )
+
+    def scan(self, root_path: Path) -> Iterator[os.DirEntry]:
+        """Scan a directory tree and yield all entries in completion order.
+
+        Spawns worker threads that pull directories from a shared queue.
+        Results are yielded as they become available. Once all workers finish,
+        any remaining buffered results are drained and yielded.
+
+        Args:
+            root_path: Root directory to begin scanning from.
+
+        Yields:
+            os.DirEntry: Every entry found in the tree that was not pruned
+                by dirs or max_depth. Entries are yielded in worker
+                completion order, not filesystem order.
+        """
+        # Queue holds (path, depth) tuples. Root starts at depth 0.
+        dir_queue: Queue[tuple[Path, int]] = Queue()
+        result_queue: Queue[os.DirEntry] = Queue()
+        state = _ScanState()
+
+        state.add(1)
+        dir_queue.put((root_path, 0))
+
+        logger.info(
+            f"Starting scan: {root_path} "
+            f"({self._max_workers} workers, max_depth={self._max_depth!r})"
+        )
+
+        with ThreadPoolExecutor(max_workers=self._max_workers) as executor:
+            for _ in range(self._max_workers):
+                executor.submit(self._worker, dir_queue, result_queue, state)
+
+            while not state.is_done:
+                try:
+                    yield result_queue.get(timeout=0.05)
+                except Empty:
+                    continue
+
+        # Executor has joined — all workers are done. Drain any remaining results.
+        drained = 0
+        while not result_queue.empty():
+            try:
+                yield result_queue.get_nowait()
+                drained += 1
+            except Empty:
+                break
+
+        if drained:
+            logger.debug(f"Drained {drained} remaining entries after workers finished")
+
+        logger.info(f"Scan complete: {root_path}")
+
+    def _worker(
+        self,
+        dir_queue: Queue,
+        result_queue: Queue,
+        state: _ScanState,
+    ) -> None:
+        """Worker loop that pulls directories from the queue and scans them.
+
+        Runs until the scan state signals completion. For each directory
+        dequeued, scans its contents and re-enqueues any discovered
+        subdirectories for other workers to pick up, unless the depth limit
+        has been reached.
+
+        New subdirectories are registered with state before being enqueued
+        to prevent a race condition where all workers complete before new
+        work is visible to the state counter.
+
+        Args:
+            dir_queue: Shared queue of (directory, depth) tuples to scan.
+            result_queue: Shared queue to put discovered entries into.
+            state: Shared scan state used to track pending work.
+        """
+        while not state.is_done:
+            try:
+                current_dir, depth = dir_queue.get(timeout=0.05)
+            except Empty:
+                continue
+
+            logger.debug(f"Worker picked up: {current_dir} (depth={depth})")
+            new_dirs = self._scan_dir(current_dir, depth, result_queue)
+
+            # Register new work BEFORE completing current — prevents false done signal.
+            state.add(len(new_dirs))
+            for d, child_depth in new_dirs:
+                dir_queue.put((d, child_depth))
+            state.complete()
+
+    def _scan_dir(
+        self, dir_path: Path, depth: int, result_queue: Queue
+    ) -> list[tuple[Path, int]]:
+        """Scan a single directory and collect results and subdirectories.
+
+        All entries are placed into result_queue. Subdirectories that are not
+        filtered by dirs are returned for re-enqueueing, unless max_depth has
+        been reached.
+
+        Args:
+            dir_path: Path to the directory to scan.
+            depth: Depth of dir_path relative to the scan root (root = 0).
+            result_queue: Queue to put discovered DirEntry objects into.
+
+        Returns:
+            List of (subdirectory path, child depth) tuples to enqueue for
+            further scanning. Empty when at or beyond max_depth.
+        """
+        new_dirs: list[tuple[Path, int]] = []
+        at_depth_limit = self._max_depth is not None and depth >= self._max_depth
+        logger.debug(f"Scanning: {dir_path} (depth={depth}, limit={self._max_depth!r})")
+
+        try:
+            with os.scandir(dir_path) as it:
+                for entry in it:
+                    try:
+                        is_dir = entry.is_dir(follow_symlinks=False)
+                    except OSError as e:
+                        logger.warning(f"Skipping entry {entry.path}: {e}")
+                        continue
+
+                    result_queue.put(entry)
+
+                    if is_dir:
+                        if (
+                            self._filter_mode == FilterMode.IGNORE
+                            and entry.name in self._dirs
+                        ):
+                            logger.debug(f"Skipping ignored directory: {entry.path}")
+                        elif (
+                            self._filter_mode == FilterMode.INCLUDE
+                            and entry.name not in self._dirs
+                        ):
+                            logger.debug(
+                                f"Skipping directory not in include list: {entry.path}"
+                            )
+                        elif at_depth_limit:
+                            logger.debug(
+                                f"Depth limit reached ({depth}/{self._max_depth}), "
+                                f"not descending into: {entry.path}"
+                            )
+                        else:
+                            logger.debug(f"Queuing subdirectory: {entry.path}")
+                            new_dirs.append((Path(entry.path), depth + 1))
+
+        except PermissionError:
+            logger.warning(f"Permission denied: {dir_path}")
+        except OSError as e:
+            logger.warning(f"Cannot scan {dir_path}: {e}")
+
+        logger.debug(
+            f"Finished scanning: {dir_path} — found {len(new_dirs)} subdirectories"
+        )
+        return new_dirs

dscanpy-0.1.0/src/dscan/filter.py

@@ -0,0 +1,140 @@
+import fnmatch
+import logging
+import os
+from collections.abc import Callable, Iterator
+from typing import Literal
+
+from enum import Enum
+
+
+class FilterMode(Enum):
+    IGNORE = "ignore"
+    INCLUDE = "include"
+
+
+logger = logging.getLogger(__name__)
+
+
+class ScanFilter:
+    """Post-scan filter that wraps a DriveScanner iterator.
+
+    Operates on yielded entries only — traversal behaviour (depth,
+    which dirs to recurse into) is controlled by DriveScanner itself.
+
+    Example:
+        scanner = DriveScanner(max_depth=3)
+
+        # Only yield .pdf and .pptx files:
+        f = ScanFilter(extensions={".pdf", ".pptx"})
+
+        # Yield everything except .pdf:
+        f = ScanFilter(extensions={".pdf"}, extensions_mode="ignore")
+
+        # Yield entries whose name does NOT match a pattern:
+        f = ScanFilter(name_pattern=".*", name_pattern_mode="ignore")
+
+        for entry in f.apply(scanner.scan(root)):
+            print(entry.path)
+    """
+
+    def __init__(
+        self,
+        only_files: bool = False,
+        only_dirs: bool = False,
+        extensions: set[str] | None = None,
+        extensions_mode: FilterMode = FilterMode.INCLUDE,
+        name_pattern: str | None = None,
+        name_pattern_mode: FilterMode = FilterMode.INCLUDE,
+        custom: Callable[[os.DirEntry], bool] | None = None,
+    ) -> None:
+        """Initialises the ScanFilter.
+
+        Args:
+            only_files: If True, only yield file entries.
+            only_dirs: If True, only yield directory entries.
+                Mutually exclusive with only_files.
+            extensions: Set of file extensions to filter on (e.g. {".pdf", ".py"}).
+                Has no effect on directory entries. Case-insensitive.
+            extensions_mode: Controls how extensions is applied:
+                - FilterMode.INCLUDE — only yield files whose extension is in extensions.
+                - FilterMode.IGNORE  — skip files whose extension is in extensions.
+            name_pattern: A glob pattern matched against entry.name (e.g. ".*", "*.min.*").
+            name_pattern_mode: Controls how name_pattern is applied:
+                - FilterMode.INCLUDE — only yield entries whose name matches the pattern.
+                - FilterMode.IGNORE  — skip entries whose name matches the pattern.
+            custom: Optional callable that receives a DirEntry and returns True to
+                keep the entry, False to drop it. Applied last, after all other filters.
+        """
+        if only_files and only_dirs:
+            raise ValueError("only_files and only_dirs are mutually exclusive")
+
+        self._only_files = only_files
+        self._only_dirs = only_dirs
+        self._extensions = {ext.lower() for ext in extensions} if extensions else None
+        self._extensions_mode = extensions_mode
+        self._name_pattern = name_pattern
+        self._name_pattern_mode = name_pattern_mode
+        self._custom = custom
+
+        logger.debug(
+            f"ScanFilter initialised — "
+            f"only_files={self._only_files}, "
+            f"only_dirs={self._only_dirs}, "
+            f"extensions={self._extensions}, "
+            f"extensions_mode={self._extensions_mode!r}, "
+            f"name_pattern={self._name_pattern!r}, "
+            f"name_pattern_mode={self._name_pattern_mode!r}, "
+            f"custom={'provided' if custom else 'None'}"
+        )
+
+    def apply(self, entries: Iterator[os.DirEntry]) -> Iterator[os.DirEntry]:
+        """Apply all configured filters to an entry iterator.
+
+        Args:
+            entries: Iterator of DirEntry objects, typically from DriveScanner.scan().
+
+        Yields:
+            os.DirEntry: Entries that pass all active filters.
+        """
+        for entry in entries:
+            if self._only_files and not entry.is_file(follow_symlinks=False):
+                logger.debug(f"ScanFilter: skipping non-file: {entry.path}")
+                continue
+            if self._only_dirs and not entry.is_dir(follow_symlinks=False):
+                logger.debug(f"ScanFilter: skipping non-directory: {entry.path}")
+                continue
+
+            if self._extensions and not entry.is_dir(follow_symlinks=False):
+                ext = os.path.splitext(entry.name)[1].lower()
+                match = ext in self._extensions
+                if self._extensions_mode == FilterMode.INCLUDE and not match:
+                    logger.debug(
+                        f"ScanFilter: extension {ext!r} not in include list: {entry.path}"
+                    )
+                    continue
+                if self._extensions_mode == FilterMode.IGNORE and match:
+                    logger.debug(
+                        f"ScanFilter: extension {ext!r} in ignore list: {entry.path}"
+                    )
+                    continue
+
+            if self._name_pattern:
+                match = fnmatch.fnmatch(entry.name, self._name_pattern)
+                if self._name_pattern_mode == FilterMode.INCLUDE and not match:
+                    logger.debug(
+                        f"ScanFilter: name {entry.name!r} doesn't match include "
+                        f"pattern {self._name_pattern!r}: {entry.path}"
+                    )
+                    continue
+                if self._name_pattern_mode == FilterMode.IGNORE and match:
+                    logger.debug(
+                        f"ScanFilter: name {entry.name!r} matches ignore "
+                        f"pattern {self._name_pattern!r}: {entry.path}"
+                    )
+                    continue
+
+            if self._custom and not self._custom(entry):
+                logger.debug(f"ScanFilter: rejected by custom filter: {entry.path}")
+                continue
+
+            yield entry

dscanpy-0.1.0/src/dscan/models.py

@@ -0,0 +1,50 @@
+from datetime import datetime
+import os
+
+from dataclasses import dataclass
+from datetime import datetime
+
+
+@dataclass(slots=True, frozen=True)
+class FileEntry:
+    name: str
+    extension: str | None
+    path: str
+    dir_path: str
+    size: int
+    created_at: datetime
+    modified_at: datetime
+
+    @classmethod
+    def from_dir_entry(cls, entry: os.DirEntry) -> "FileEntry":
+        stat = entry.stat(follow_symlinks=False)
+        name, ext = os.path.splitext(entry.name)
+        return cls(
+            name=name,
+            extension=ext[1:].lower() if ext else None,
+            path=entry.path,
+            dir_path=os.path.dirname(entry.path),
+            size=stat.st_size,
+            created_at=datetime.fromtimestamp(stat.st_ctime),
+            modified_at=datetime.fromtimestamp(stat.st_mtime),
+        )
+
+
+@dataclass(slots=True, frozen=True)
+class DirectoryEntry:
+    name: str
+    path: str
+    parent_path: str
+    created_at: datetime
+    modified_at: datetime
+
+    @classmethod
+    def from_dir_entry(cls, entry: os.DirEntry) -> "DirectoryEntry":
+        stat = entry.stat(follow_symlinks=False)
+        return cls(
+            name=entry.name,
+            path=entry.path,
+            parent_path=os.path.dirname(entry.path),
+            created_at=datetime.fromtimestamp(stat.st_ctime),
+            modified_at=datetime.fromtimestamp(stat.st_mtime),
+        )