ds_crawler 1.0.1__tar.gz

ds_crawler-1.0.1/.github/workflows/workflow.yml

@@ -0,0 +1,30 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - "v*" # This triggers the workflow only when you push a tag starting with 'v'
+
+jobs:
+  build-and-publish:
+    name: Build and Publish
+    runs-on: ubuntu-latest
+    environment: pypi # This matches the environment we set up in Step 2
+
+    permissions:
+      id-token: write # CRITICAL: This is what allows OIDC (passwordless) auth to PyPI
+      contents: read # Required to check out the code
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5 # The official Astral action
+
+      - name: Build package
+        run: uv build
+
+      - name: Publish to PyPI
+        # uv publish automatically detects it is in GitHub Actions and uses OIDC
+        run: uv publish

ds_crawler-1.0.1/.gitignore

@@ -0,0 +1,7 @@
+__pycache__
+.DS_Store
+.vscode/
+.claude/
+/.pytest_cache
+SHOULD_MATCH
+/cluster/

ds_crawler-1.0.1/PKG-INFO

@@ -0,0 +1,10 @@
+Metadata-Version: 2.4
+Name: ds_crawler
+Version: 1.0.1
+Summary: Regex-based dataset metadata crawler and indexer
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: tqdm; extra == 'dev'
+Provides-Extra: progress
+Requires-Dist: tqdm; extra == 'progress'

ds_crawler-1.0.1/README.md

@@ -0,0 +1,398 @@
+# ds-crawler
+
+Regex-based dataset metadata crawler and indexer. Extracts structured
+identifiers from file paths, organises them into a hierarchical index
+(`output.json`), and provides utilities for aligning, copying, splitting
+and writing multi-modal datasets.
+
+```
+pip install .                 # core
+pip install ".[progress]"     # with tqdm progress bars
+pip install ".[dev]"          # with pytest + tqdm
+uv pip install "ds-crawler @ git+https://github.com/d-rothen/ds-crawler"
+```
+
+Requires Python >= 3.9. No runtime dependencies.
+
+---
+
+## Quick start
+
+### Index a dataset
+
+Every dataset needs a small JSON config (`ds-crawler.json`) that tells
+the crawler how to extract file IDs and (optionally) a hierarchy from
+paths. Place it either inside `.ds_crawler/ds-crawler.json` at the
+dataset root, or pass the config dict directly.
+
+```python
+from ds_crawler import index_dataset_from_path
+
+# Reads ds-crawler.json from inside the dataset, returns an output dict
+output = index_dataset_from_path("/data/rgb", save_index=True)
+```
+
+### Align modalities by ID
+
+```python
+from ds_crawler import align_datasets
+
+aligned = align_datasets(
+    {"modality": "rgb",   "source": "/data/rgb"},
+    {"modality": "depth", "source": "/data/depth"},
+)
+
+for file_id, mods in aligned.items():
+    if "rgb" in mods and "depth" in mods:
+        print(mods["rgb"]["path"], mods["depth"]["path"])
+```
+
+### Split into train / val
+
+```python
+from ds_crawler import split_datasets
+
+result = split_datasets(
+    source_paths=["/data/rgb", "/data/depth"],
+    suffixes=["train", "val"],
+    ratios=[80, 20],
+    seed=42,
+)
+```
+
+### Write model outputs back to disk
+
+```python
+from ds_crawler import DatasetWriter
+
+writer = DatasetWriter(
+    "/output/segmentation",
+    name="segmentation",
+    type="segmentation",
+    euler_train={"used_as": "target", "modality_type": "semantic"},
+)
+
+for sample in dataloader:
+    pred = model(sample["rgb"])
+    path = writer.get_path(sample["full_id"], f"{sample['id']}.png")
+    save_image(pred, path)
+
+writer.save_index()  # writes output.json for later re-indexing
+```
+
+---
+
+## Configuration
+
+### `ds-crawler.json`
+
+Placed at `<dataset_root>/.ds_crawler/ds-crawler.json` (or passed as a
+dict). Minimal example:
+
+```json
+{
+  "name": "my_rgb",
+  "path": "/data/my_rgb",
+  "type": "rgb",
+  "id_regex": "^frame_(\\d+)\\.png$",
+  "properties": {
+    "euler_train": {
+      "used_as": "input",
+      "modality_type": "rgb"
+    }
+  }
+}
+```
+
+| Field | Required | Description |
+|---|---|---|
+| `name` | yes | Human-readable dataset name |
+| `path` | yes | Root directory or `.zip` archive |
+| `type` | yes | Semantic label for the data modality (e.g. `"rgb"`, `"depth"`) |
+| `id_regex` | yes | Regex applied to each file's relative path. Capture groups form the file ID (joined by `id_regex_join_char`). |
+| `properties.euler_train.used_as` | yes | `"input"`, `"target"`, or `"condition"` |
+| `properties.euler_train.modality_type` | yes | Identifier token (e.g. `"rgb"`, `"depth"`) |
+| `hierarchy_regex` | no | Regex with named groups to build a hierarchy tree |
+| `named_capture_group_value_separator` | no | Character joining group name and value in hierarchy keys (default `":"`) |
+| `basename_regex` | no | Regex applied to basename only; properties stored per file |
+| `path_regex` | no | Regex applied to full relative path; properties stored per file |
+| `path_filters` | no | Path-level include/exclude rules (regex and/or term whitelist/blacklist) |
+| `intrinsics_regex` | no | Regex matching camera intrinsics files |
+| `extrinsics_regex` | no | Regex matching camera extrinsics files |
+| `id_regex_join_char` | no | Join character for multi-group IDs (default `"+"`) |
+| `file_extensions` | no | Restrict to these extensions (e.g. `[".png", ".jpg"]`) |
+| `flat_ids_unique` | no | If `true`, IDs must be globally unique (not just within hierarchy node) |
+| `output_json` | no | Path to a pre-existing `output.json` to load instead of crawling |
+
+#### Path filters (`path_filters`)
+
+Use `path_filters` when you want to include/exclude files by relative path
+without changing your `id_regex`:
+
+```json
+"path_filters": {
+  "include_terms": ["fog"],
+  "exclude_terms": ["night"],
+  "term_match_mode": "path_segment"
+}
+```
+
+Supported keys:
+
+- `include_regex`: list of regexes, at least one must match when provided
+- `exclude_regex`: list of regexes, none may match
+- `include_terms`: term whitelist, at least one must match when provided
+- `exclude_terms`: term blacklist, none may match
+- `term_match_mode`: `"substring"` (default) or `"path_segment"`
+- `case_sensitive`: `true` (default) or `false`
+
+These filters apply to both data files and camera metadata paths
+(`intrinsics_regex` / `extrinsics_regex` matches).
+
+Example: keep only VKITTI fog frames:
+
+```json
+"path_filters": {
+  "include_terms": ["fog"],
+  "term_match_mode": "path_segment"
+}
+```
+
+Example: exclude all fog frames:
+
+```json
+"path_filters": {
+  "exclude_terms": ["fog"],
+  "term_match_mode": "path_segment"
+}
+```
+
+### Multi-dataset config
+
+For the CLI, wrap multiple dataset configs in:
+
+```json
+{
+  "datasets": [
+    { "name": "rgb",   "path": "/data/rgb",   "type": "rgb",   "id_regex": "..." },
+    { "name": "depth", "path": "/data/depth", "type": "depth", "id_regex": "..." }
+  ]
+}
+```
+
+---
+
+## Output format (`output.json`)
+
+The index produced by the crawler:
+
+```json
+{
+  "name": "my_rgb",
+  "type": "rgb",
+  "id_regex": "...",
+  "id_regex_join_char": "+",
+  "path_filters": {
+    "include_terms": ["fog"],
+    "term_match_mode": "path_segment"
+  },
+  "euler_train": { "used_as": "input", "modality_type": "rgb" },
+  "named_capture_group_value_separator": ":",
+  "dataset": {
+    "files": [
+      { "path": "frame_001.png", "id": "001", "path_properties": {}, "basename_properties": {} }
+    ],
+    "children": {
+      "scene:Scene01": {
+        "files": [ ... ],
+        "children": { ... }
+      }
+    }
+  }
+}
+```
+
+`dataset` is a recursive node: each node has `files` (leaf entries) and
+`children` (named sub-nodes). Hierarchy keys follow the pattern
+`<group_name><separator><value>` (e.g. `scene:Scene01`).
+
+---
+
+## CLI
+
+```
+ds-crawler CONFIG [OPTIONS]
+```
+
+| Flag | Description |
+|---|---|
+| `-o, --output PATH` | Write a single combined output file (otherwise writes per-dataset) |
+| `-w, --workdir PATH` | Prepend to relative dataset paths |
+| `-s, --strict` | Abort on duplicate IDs or >20% regex misses |
+| `--sample N` | Keep every Nth matched file |
+| `--match-index PATH` | Only include file IDs present in this output.json |
+| `-v, --verbose` | Log every skipped file |
+
+---
+
+## Python API
+
+All public symbols are re-exported from the top-level `ds_crawler`
+package. They live in four submodules:
+
+### Indexing (`ds_crawler.parser`)
+
+#### `index_dataset(config, *, strict, save_index, sample, match_index) -> dict`
+
+Index a single dataset from a config dict.
+
+#### `index_dataset_from_path(path, *, strict, save_index, force_reindex, sample, match_index) -> dict`
+
+Index a dataset by path, reading `ds-crawler.json` from the dataset root.
+Returns a cached `output.json` when available (unless `force_reindex=True`).
+
+#### `index_dataset_from_files(config, files, *, base_path, strict, sample, match_index) -> dict`
+
+Index from pre-collected file paths (useful when files aren't on the
+local filesystem).
+
+#### `DatasetParser(config, *, strict, sample, match_index)`
+
+Lower-level class wrapping the full `Config` object. Methods:
+
+- `parse_all() -> list[dict]`
+- `parse_dataset(ds_config, ...) -> dict`
+- `parse_dataset_from_files(ds_config, files, ...) -> dict`
+- `write_output(output_path)`
+- `write_outputs_per_dataset(filename="output.json") -> list[Path]`
+
+---
+
+### Traversal & filtering (`ds_crawler.traversal`)
+
+#### `get_files(output_json) -> list[str]`
+
+Flat list of every file path in an output dict (or list of output dicts).
+
+#### `collect_qualified_ids(output_json) -> set[tuple[str, ...]]`
+
+Set of `(*hierarchy_keys, file_id)` tuples. Qualified IDs distinguish
+files that share the same raw ID but live at different hierarchy levels.
+
+#### `filter_index_by_qualified_ids(output_json, qualified_ids) -> dict`
+
+Return a pruned copy of the output dict keeping only the given IDs.
+
+#### `split_qualified_ids(qualified_ids, ratios, *, seed=None) -> list[set]`
+
+Partition qualified IDs by percentage (e.g. `[80, 20]`). Deterministic
+when `seed` is `None` (sorted order) or fixed.
+
+---
+
+### Operations (`ds_crawler.operations`)
+
+#### `align_datasets(*args) -> dict[str, dict[str, dict]]`
+
+Align multiple modalities by file ID. Each positional argument is a dict
+with `"modality"` (label) and `"source"` (path or output dict). Returns
+`{file_id: {modality: file_entry, ...}, ...}`.
+
+#### `copy_dataset(input_path, output_path, *, index=None, sample=None) -> dict`
+
+Copy dataset files to a new location (directory or `.zip`), preserving
+structure. Returns `{"copied": int, "missing": int, "missing_files": [...]}`.
+
+#### `split_dataset(source_path, ratios, target_paths, *, qualified_ids, seed) -> dict`
+
+Split a single dataset into multiple targets by percentage.
+
+#### `split_datasets(source_paths, suffixes, ratios, *, seed) -> dict`
+
+Split multiple aligned datasets using their common ID intersection.
+Target paths are derived by appending the suffix
+(`/data/rgb` + `"train"` -> `/data/rgb_train`).
+
+---
+
+### Writer (`ds_crawler.writer`)
+
+#### `DatasetWriter(root, *, name, type, euler_train, separator=":", **properties)`
+
+Stateful helper that turns `(full_id, basename)` pairs into filesystem
+paths while accumulating an `output.json`-compatible index.
+
+| Method | Description |
+|---|---|
+| `get_path(full_id, basename, *, source_meta=None) -> Path` | Register a file and get the absolute path to write to. Directories are created automatically. |
+| `build_output() -> dict` | Return the accumulated index as an output dict. |
+| `save_index(filename="output.json") -> Path` | Persist the index to `<root>/.ds_crawler/<filename>`. |
+
+`full_id` follows the format produced by euler-loading:
+`/scene:Scene01/camera:Cam0/scene-Scene01+camera-Cam0+frame-00001`
+where each `/key:value` segment maps to a directory level and the final
+component is the ds-crawler file ID.
+
+---
+
+### Validation (`ds_crawler.validation`)
+
+#### `validate_crawler_config(config, workdir=None) -> DatasetConfig`
+
+Validate a `ds-crawler.json` dict. Raises `ValueError` on failure.
+
+#### `validate_output(output) -> dict`
+
+Validate an `output.json` object (single dict or list). Raises
+`ValueError` on failure.
+
+#### `validate_dataset(path) -> dict`
+
+Check a dataset path for valid metadata files. Returns
+`{"path", "has_config", "has_output", "config", "output"}`.
+
+---
+
+### Schema (`ds_crawler.schema`)
+
+#### `DatasetDescriptor(name, path, type, properties={})`
+
+Minimal dataset description dataclass. Class methods:
+
+- `from_output(data, path) -> DatasetDescriptor`
+- `from_output_file(path, dataset_root) -> list[DatasetDescriptor]`
+
+---
+
+### Config (`ds_crawler.config`)
+
+#### `DatasetConfig`
+
+Full dataset configuration (extends `DatasetDescriptor` with regex
+fields). Usually created via `DatasetConfig.from_dict(data, workdir)` or
+`load_dataset_config(data, workdir)`.
+
+#### `Config(datasets: list[DatasetConfig])`
+
+Container loaded with `Config.from_file(path, workdir)`.
+
+---
+
+## ZIP support
+
+All operations work transparently with `.zip` archives. Paths like
+`/data/dataset.zip` are handled the same as directories: the crawler
+reads file listings from the archive, `copy_dataset` writes into a new
+archive, and `save_index` / `write_outputs_per_dataset` embed
+`output.json` inside the zip.
+
+---
+
+## Examples
+
+See the [`examples/`](examples/) directory:
+
+- **`example.py`** -- index a dataset, match against an existing index,
+  and copy a subset.
+- **`sample_realdrivesim.py`** -- subsample a multi-modal RealDriveSim
+  dataset (RGB, depth, segmentation) preserving cross-modality alignment.

ds_crawler-1.0.1/config_example.json

@@ -0,0 +1,75 @@
+{
+  "datasets": [
+    {
+      "name": "VKITTI2",
+      "path": "/path/to/vkitti2/rgb",
+      "type": "rgb",
+      "path_regex": "^(?P<scene>Scene\\d+)/(?P<variation>[^/]+)/frames/rgb/(?P<camera>Camera_\\d+)/",
+      "basename_regex": "^rgb_(?P<frame>\\d+)\\.(?P<ext>jpg|png)$",
+      "named_capture_group_value_separator": ":",
+      "intrinsics_regex": "^(?P<scene>Scene\\d+)/(?P<variation>[^/]+)/intrinsics/(?P<camera>Camera_\\d+)_intrinsics\\.txt$",
+      "hierarchy_regex": "^(?P<scene>Scene\\d+)/(?P<variation>[^/]+)/frames/rgb/(?P<camera>Camera_\\d+)/rgb_(?P<frame>\\d+)\\.png$",
+      "id_regex": "^(?P<scene>Scene\\d+)/(?P<variation>[^/]+)/frames/rgb/(?P<camera>Camera_\\d+)/rgb_(?P<frame>\\d+)\\.(?:jpg|png)$",
+      "properties": {
+        "gt": true,
+        "baseline": true,
+        "euler_train": {
+          "used_as": "target",
+          "slot": "demo.target.rgb",
+          "modality_type": "rgb"
+        },
+        "meta": {
+          "range": [0, 255]
+        },
+        "specifier": "whatever",
+        "some_other_prop": 1234566
+      }
+    },
+    {
+      "name": "VKITTI2_depth",
+      "path": "/path/to/vkitti2/depth",
+      "type": "depth",
+      "path_regex": "^(?P<scene>Scene\\d+)/(?P<variation>[^/]+)/frames/depth/(?P<camera>Camera_\\d+)/",
+      "basename_regex": "^depth_(?P<frame>\\d+)\\.(?P<ext>png)$",
+      "named_capture_group_value_separator": ":",
+      "hierarchy_regex": "^(?P<scene>Scene\\d+)/(?P<variation>[^/]+)/frames/depth/(?P<camera>Camera_\\d+)/depth_(?P<frame>\\d+)\\.png$",
+      "id_regex": "^(?P<scene>Scene\\d+)/(?P<variation>[^/]+)/frames/depth/(?P<camera>Camera_\\d+)/depth_(?P<frame>\\d+)\\.png$",
+      "properties": {
+        "gt": true,
+        "euler_train": {
+          "used_as": "target",
+          "slot": "demo.target.depth",
+          "modality_type": "depth"
+        },
+        "meta": {
+          "radial_depth": false,
+          "scale_to_meters": 1.0,
+          "range": [0, 65535]
+        }
+      }
+    },
+    {
+      "name": "predicted_depth",
+      "path": "/path/to/predicted_depth",
+      "type": "depth",
+      "file_extensions": [".png", ".npy"],
+      "basename_regex": "^(?P<frame>\\d+)_pred\\.(?P<ext>png|npy)$",
+      "named_capture_group_value_separator": ":",
+      "id_regex": "^(?P<frame>\\d+)_pred\\.png$",
+      "hierarchy_regex": "^(?P<frame>\\d+)_pred\\.(?:png|npy)$",
+      "properties": {
+        "gt": false,
+        "euler_train": {
+          "used_as": "input",
+          "slot": "demo.input.depth",
+          "modality_type": "depth"
+        },
+        "meta": {
+          "radial_depth": false,
+          "scale_to_meters": 1.0,
+          "range": [0, 65535]
+        }
+      }
+    }
+  ]
+}

ds_crawler-1.0.1/ds_crawler/__init__.py

@@ -0,0 +1,50 @@
+"""Dataset crawler package."""
+
+from .schema import DatasetDescriptor
+from .config import Config, DatasetConfig, load_dataset_config
+from .validation import validate_crawler_config, validate_dataset, validate_output
+from .parser import (
+    DatasetParser,
+    index_dataset,
+    index_dataset_from_files,
+    index_dataset_from_path,
+)
+from .traversal import (
+    collect_qualified_ids,
+    filter_index_by_qualified_ids,
+    get_files,
+    split_qualified_ids,
+)
+from .operations import (
+    align_datasets,
+    copy_dataset,
+    extract_datasets,
+    split_dataset,
+    split_datasets,
+)
+from .writer import DatasetWriter, ZipDatasetWriter
+
+__all__ = [
+    "DatasetDescriptor",
+    "DatasetWriter",
+    "ZipDatasetWriter",
+    "Config",
+    "DatasetConfig",
+    "DatasetParser",
+    "align_datasets",
+    "collect_qualified_ids",
+    "copy_dataset",
+    "extract_datasets",
+    "filter_index_by_qualified_ids",
+    "get_files",
+    "index_dataset",
+    "index_dataset_from_files",
+    "index_dataset_from_path",
+    "load_dataset_config",
+    "split_dataset",
+    "split_datasets",
+    "split_qualified_ids",
+    "validate_crawler_config",
+    "validate_dataset",
+    "validate_output",
+]

ds_crawler-1.0.1/ds_crawler/cli.py

@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""Dataset crawler main entry point."""
+
+import argparse
+import json
+import logging
+import sys
+from pathlib import Path
+
+from ds_crawler.config import Config
+from ds_crawler.parser import DatasetParser
+
+
+def setup_logging(verbose: bool) -> None:
+    """Configure logging with immediate flushing for cluster compatibility."""
+
+    class FlushingHandler(logging.StreamHandler):
+        def emit(self, record):
+            super().emit(record)
+            self.flush()
+
+    level = logging.DEBUG if verbose else logging.INFO
+    handler = FlushingHandler(sys.stderr)
+    handler.setLevel(level)
+    handler.setFormatter(logging.Formatter("%(levelname)s: %(message)s"))
+
+    logging.root.handlers = []
+    logging.root.addHandler(handler)
+    logging.root.setLevel(level)
+
+
+def main() -> int:
+    """Main entry point."""
+    parser = argparse.ArgumentParser(
+        description="Crawl datasets and extract metadata based on configuration."
+    )
+    parser.add_argument(
+        "config",
+        type=Path,
+        help="Path to the configuration JSON file",
+    )
+    parser.add_argument(
+        "-o",
+        "--output",
+        type=Path,
+        default=None,
+        help="Single output JSON file path. If not specified, writes output.json to each dataset's root folder.",
+    )
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        action="store_true",
+        help="Enable verbose output (show each skipped file)",
+    )
+    parser.add_argument(
+        "-w",
+        "--workdir",
+        type=Path,
+        default=None,
+        help="Working directory to prepend to dataset paths.",
+    )
+    parser.add_argument(
+        "-s",
+        "--strict",
+        action="store_true",
+        help="Strict mode: abort on duplicate IDs or excessive regex misses.",
+    )
+    parser.add_argument(
+        "--sample",
+        type=int,
+        default=None,
+        metavar="N",
+        help="Keep every Nth regex-matched file (deterministic subsampling).",
+    )
+    parser.add_argument(
+        "--match-index",
+        type=Path,
+        default=None,
+        metavar="PATH",
+        help="Path to an output.json whose file IDs are used as a filter.",
+    )
+
+    args = parser.parse_args()
+
+    setup_logging(args.verbose)
+
+    try:
+        config = Config.from_file(args.config, workdir=args.workdir)
+    except FileNotFoundError as e:
+        print(f"Error: {e}", file=sys.stderr)
+        return 1
+    except ValueError as e:
+        print(f"Configuration error: {e}", file=sys.stderr)
+        return 1
+
+    match_index = None
+    if args.match_index is not None:
+        try:
+            with open(args.match_index) as f:
+                match_index = json.load(f)
+        except (FileNotFoundError, json.JSONDecodeError) as e:
+            print(f"Error loading match-index: {e}", file=sys.stderr)
+            return 1
+
+    parser_instance = DatasetParser(
+        config,
+        strict=args.strict,
+        sample=args.sample,
+        match_index=match_index,
+    )
+
+    logger = logging.getLogger(__name__)
+
+    try:
+        if args.output:
+            parser_instance.write_output(args.output)
+            logger.info(f"Output written to: {args.output}")
+        else:
+            output_paths = parser_instance.write_outputs_per_dataset()
+            for path in output_paths:
+                logger.info(f"Output written to: {path}")
+    except Exception as e:
+        print(f"Error processing datasets: {e}", file=sys.stderr)
+        return 1
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())