evaluators 1.0.3__tar.gz → 2.0.0__tar.gz

evaluators-2.0.0/PKG-INFO

@@ -0,0 +1,279 @@
+Metadata-Version: 2.3
+Name: evaluators
+Version: 2.0.0
+Summary: Various scene understanding and perception evaluation metrics.
+Author: Kurt Stolle
+Author-email: Kurt Stolle <kurt@computer.org>
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Requires-Dist: torch>2.7
+Requires-Dist: torchvision
+Requires-Dist: torchmetrics
+Requires-Dist: tensordict>=0.11.0
+Requires-Dist: numpy>=2.0.0
+Requires-Dist: matplotlib>=3.9.0
+Requires-Dist: rich>=13.9.4
+Requires-Dist: lmdb
+Requires-Dist: tifffile
+Requires-Dist: pydantic
+Requires-Dist: opencv-python>=4.13.0.92
+Requires-Python: >=3.13, <3.14
+Description-Content-Type: text/markdown
+
+# Scalable Distributed Evaluation for Computer Vision
+
+[![PyPI](https://img.shields.io/pypi/v/evaluators)](https://pypi.org/project/evaluators/)
+[![Python Version](https://img.shields.io/pypi/pyversions/evaluators)](https://pypi.org/project/evaluators/)
+[![License](https://img.shields.io/pypi/l/evaluators)](https://github.com/khwstolle/evaluators/blob/main/LICENSE)
+[![Build Status](https://img.shields.io/github/actions/workflow/status/khwstolle/evaluators/ci.yml)](https://github.com/khwstolle/evaluators/actions)
+
+Evaluators is a high-throughput evaluation framework designed for large-scale computer vision research. It specializes in handling video tasks by decoupling inference I/O from metric computation.
+
+This architecture enables offline evaluation workflows: models stream predictions to efficient storage backends (Memory Map or LMDB) during inference, and metrics are computed in a decoupled stage using distributed map-reduce logic. This approach prevents CPU-bound metric calculation from throttling GPU inference.
+
+The key features are:
+
+- **Zero-overhead inference.**  
+  Writes predictions to disk using non-blocking I/O, allowing the training loop to run at full GPU utilization.
+
+- **Distributed by design.**  
+  Automatically handles synchronization across multiple nodes and GPUs using `torchmetrics` and custom scheduling logic.
+
+- **Explicit Memory Schemas.**
+  Uses Pydantic-based schemas to define data formats and encodings (PNG, TIFF, Raw) up front, ensuring type safety and storage efficiency.
+
+- **Lazy loading.**  
+  Supports referencing ground truth data from disk rather than duplicating it in memory caches, enabling evaluation of terabyte-scale datasets.
+
+- **Multi-domain.**
+  Includes verified implementations for:
+  - Segmentation: Panoptic quality (PQ), semantic mIoU.
+  - DVPS: Depth-aware video panoptic quality (DVPQ).
+  - Depth: Eigen et al. metrics (AbsRel, RMSE).
+
+- **CLI.**
+  A command-line interface to inspect, index, and query saved inference results.
+
+---
+
+## Installation
+
+```shell
+pip install evaluators
+```
+
+## Quick start
+
+### Python API
+
+The core abstractions are `MetricStream` (for writing) and `run_offline_evaluation` (for computing).
+
+#### Step 1: Inference (online)
+
+```python
+import torch
+from evaluators import MetricStream, MemorySchema, TensorField, DynamicTemporalWriter
+from evaluators.metrics.domain.segmentation import SemanticMetric
+```
+
+##### 1. Configure metrics and schema
+
+```python
+### Define the source for ground truth (lazy loading)
+
+dataset = CityscapesDataset(...)
+metric = SemanticMetric(
+    num_classes=19,
+    target_source=dataset, 
+)
+
+### Define the explicit memory schema
+
+schema = MemorySchema(fields={
+    "sem_seg": TensorField(dtype="int64", shape=(1024, 2048)),
+    "sequence_id": TensorField(dtype="int64", shape=()),
+    "frame_index": TensorField(dtype="int64", shape=()),
+})
+```
+
+##### 2. Initialize stream and writer
+
+```python
+# Create a writer (backend)
+writer = DynamicTemporalWriter(output_dir="./inference_cache/stream_1", schema=schema)
+
+# Create a stream and bind the writer
+stream = MetricStream(
+    metrics=[metric],
+    name="semantic",
+    schema=schema
+)
+stream.bind(writer)
+```
+
+##### 3. Run inference loop
+
+```python
+for batch in dataloader: # Model forward pass
+    preds = model(batch["image"])
+
+    # Push to stream (non-blocking)
+    stream.update(
+        batch={
+            "sem_seg": preds,
+            "sequence_id": batch["sequence_id"],
+            "frame_index": batch["frame_index"]
+        }
+    )
+    
+# Finalize
+writer.close()
+```
+
+    > Note: `sequence_id` and `frame_index` must be `torch.int64`. 
+
+##### 4. Finalize and compute
+
+```python
+from evaluators import run_offline_evaluation
+
+# Syncs workers, builds catalog, and runs metrics
+results = run_offline_evaluation(
+    metrics=[metric],
+    artifact_dir="./inference_cache/semantic"
+)
+print(results["SemanticMetric"]["mIoU"])
+```
+
+#### Step 2: Re-evaluation (offline)
+
+Because predictions are persisted, metrics can be re-calculated or added without re-running the model.
+
+```python
+# Run evaluation on existing artifacts
+results = run_offline_evaluation(
+    metrics=[new_metric],
+    artifact_dir="./inference_cache/semantic"
+)
+```
+
+### CLI tools
+
+The library includes a CLI for managing the inference cache.
+
+_List stored sequences._
+
+```shell
+evaluators memory ls ./inference_cache
+```
+
+_Inspect specific tensor shapes._
+
+```shell
+evaluators memory inspect ./inference_cache --sequence_id frankfurt_000001
+```
+
+_Export to standard PyTorch file._
+
+```shell
+evaluators memory export ./inference_cache --sequence_id frankfurt_000001 --out my_video.pt
+```
+
+---
+
+## Supported metrics
+
+### Depth estimation
+
+Implements standard error metrics (AbsRel, SqRel, RMSE, RMSElog) and threshold accuracies ($\delta < 1.25^n$).
+
+### Segmentation
+
+- _Semantic:_ Mean intersection over union (mIoU).
+- _Panoptic:_ Panoptic quality (PQ), segmentation quality (SQ), recognition quality (RQ). Supports "Thing" and "Stuff" splits.
+
+### Depth-aware video panoptic segmentation (DVPS)
+
+Implements DVPQ (Depth-aware video panoptic quality). This metric evaluates spatio-temporal consistency using sliding window tubes, gated by pixel-wise depth accuracy.
+
+---
+
+## Architecture
+
+The evaluation pipeline consists of three stages.
+
+1. **Write.**  
+   Each GPU writes predictions to locally sharded files (e.g. `.memmap` or `.lmdb`). No communication occurs.
+2. **Schedule.**  
+   A synchronization barrier is reached. The main process aggregates metadata manifests from all shards to build a Global Catalog. It partitions the workload (videos) among workers using a greedy strategy to balance duration.
+3. **Compute.**  
+   Workers iterate through their assigned Virtual Sequences. Data is streamed from disk, processed by torchmetrics, and reduced globally.
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed diagrams and design principles.
+
+---
+
+## Development
+
+This project uses modern Python tooling for dependency management and quality assurance.
+
+### Setup
+
+We use `uv` for fast dependency management.
+
+```shell
+# Install dependencies
+uv sync --all-extras
+```
+
+### Testing
+
+Tests are managed by `pytest`.
+
+```shell
+# Run all tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=evaluators
+```
+
+### Linting & Formatting
+
+We use `ruff` for all linting and formatting needs.
+
+```shell
+# Check code style
+uv run ruff check .
+
+# Format code
+uv run ruff format .
+```
+
+---
+
+## Contributing
+
+Contributions are welcome. Please ensure that:
+
+1. New features are covered by tests.
+2. Code passes all static analysis checks (`ruff`).
+3. Architecture changes are discussed in an issue first.
+
+## Acknowledgements
+
+This work was developed at the Mobile Perception Systems (MPS) lab at Eindhoven University of Technology.
+
+## License
+
+This project is licensed under the MIT License.

evaluators-2.0.0/README.md

@@ -0,0 +1,247 @@
+# Scalable Distributed Evaluation for Computer Vision
+
+[![PyPI](https://img.shields.io/pypi/v/evaluators)](https://pypi.org/project/evaluators/)
+[![Python Version](https://img.shields.io/pypi/pyversions/evaluators)](https://pypi.org/project/evaluators/)
+[![License](https://img.shields.io/pypi/l/evaluators)](https://github.com/khwstolle/evaluators/blob/main/LICENSE)
+[![Build Status](https://img.shields.io/github/actions/workflow/status/khwstolle/evaluators/ci.yml)](https://github.com/khwstolle/evaluators/actions)
+
+Evaluators is a high-throughput evaluation framework designed for large-scale computer vision research. It specializes in handling video tasks by decoupling inference I/O from metric computation.
+
+This architecture enables offline evaluation workflows: models stream predictions to efficient storage backends (Memory Map or LMDB) during inference, and metrics are computed in a decoupled stage using distributed map-reduce logic. This approach prevents CPU-bound metric calculation from throttling GPU inference.
+
+The key features are:
+
+- **Zero-overhead inference.**  
+  Writes predictions to disk using non-blocking I/O, allowing the training loop to run at full GPU utilization.
+
+- **Distributed by design.**  
+  Automatically handles synchronization across multiple nodes and GPUs using `torchmetrics` and custom scheduling logic.
+
+- **Explicit Memory Schemas.**
+  Uses Pydantic-based schemas to define data formats and encodings (PNG, TIFF, Raw) up front, ensuring type safety and storage efficiency.
+
+- **Lazy loading.**  
+  Supports referencing ground truth data from disk rather than duplicating it in memory caches, enabling evaluation of terabyte-scale datasets.
+
+- **Multi-domain.**
+  Includes verified implementations for:
+  - Segmentation: Panoptic quality (PQ), semantic mIoU.
+  - DVPS: Depth-aware video panoptic quality (DVPQ).
+  - Depth: Eigen et al. metrics (AbsRel, RMSE).
+
+- **CLI.**
+  A command-line interface to inspect, index, and query saved inference results.
+
+---
+
+## Installation
+
+```shell
+pip install evaluators
+```
+
+## Quick start
+
+### Python API
+
+The core abstractions are `MetricStream` (for writing) and `run_offline_evaluation` (for computing).
+
+#### Step 1: Inference (online)
+
+```python
+import torch
+from evaluators import MetricStream, MemorySchema, TensorField, DynamicTemporalWriter
+from evaluators.metrics.domain.segmentation import SemanticMetric
+```
+
+##### 1. Configure metrics and schema
+
+```python
+### Define the source for ground truth (lazy loading)
+
+dataset = CityscapesDataset(...)
+metric = SemanticMetric(
+    num_classes=19,
+    target_source=dataset, 
+)
+
+### Define the explicit memory schema
+
+schema = MemorySchema(fields={
+    "sem_seg": TensorField(dtype="int64", shape=(1024, 2048)),
+    "sequence_id": TensorField(dtype="int64", shape=()),
+    "frame_index": TensorField(dtype="int64", shape=()),
+})
+```
+
+##### 2. Initialize stream and writer
+
+```python
+# Create a writer (backend)
+writer = DynamicTemporalWriter(output_dir="./inference_cache/stream_1", schema=schema)
+
+# Create a stream and bind the writer
+stream = MetricStream(
+    metrics=[metric],
+    name="semantic",
+    schema=schema
+)
+stream.bind(writer)
+```
+
+##### 3. Run inference loop
+
+```python
+for batch in dataloader: # Model forward pass
+    preds = model(batch["image"])
+
+    # Push to stream (non-blocking)
+    stream.update(
+        batch={
+            "sem_seg": preds,
+            "sequence_id": batch["sequence_id"],
+            "frame_index": batch["frame_index"]
+        }
+    )
+    
+# Finalize
+writer.close()
+```
+
+    > Note: `sequence_id` and `frame_index` must be `torch.int64`. 
+
+##### 4. Finalize and compute
+
+```python
+from evaluators import run_offline_evaluation
+
+# Syncs workers, builds catalog, and runs metrics
+results = run_offline_evaluation(
+    metrics=[metric],
+    artifact_dir="./inference_cache/semantic"
+)
+print(results["SemanticMetric"]["mIoU"])
+```
+
+#### Step 2: Re-evaluation (offline)
+
+Because predictions are persisted, metrics can be re-calculated or added without re-running the model.
+
+```python
+# Run evaluation on existing artifacts
+results = run_offline_evaluation(
+    metrics=[new_metric],
+    artifact_dir="./inference_cache/semantic"
+)
+```
+
+### CLI tools
+
+The library includes a CLI for managing the inference cache.
+
+_List stored sequences._
+
+```shell
+evaluators memory ls ./inference_cache
+```
+
+_Inspect specific tensor shapes._
+
+```shell
+evaluators memory inspect ./inference_cache --sequence_id frankfurt_000001
+```
+
+_Export to standard PyTorch file._
+
+```shell
+evaluators memory export ./inference_cache --sequence_id frankfurt_000001 --out my_video.pt
+```
+
+---
+
+## Supported metrics
+
+### Depth estimation
+
+Implements standard error metrics (AbsRel, SqRel, RMSE, RMSElog) and threshold accuracies ($\delta < 1.25^n$).
+
+### Segmentation
+
+- _Semantic:_ Mean intersection over union (mIoU).
+- _Panoptic:_ Panoptic quality (PQ), segmentation quality (SQ), recognition quality (RQ). Supports "Thing" and "Stuff" splits.
+
+### Depth-aware video panoptic segmentation (DVPS)
+
+Implements DVPQ (Depth-aware video panoptic quality). This metric evaluates spatio-temporal consistency using sliding window tubes, gated by pixel-wise depth accuracy.
+
+---
+
+## Architecture
+
+The evaluation pipeline consists of three stages.
+
+1. **Write.**  
+   Each GPU writes predictions to locally sharded files (e.g. `.memmap` or `.lmdb`). No communication occurs.
+2. **Schedule.**  
+   A synchronization barrier is reached. The main process aggregates metadata manifests from all shards to build a Global Catalog. It partitions the workload (videos) among workers using a greedy strategy to balance duration.
+3. **Compute.**  
+   Workers iterate through their assigned Virtual Sequences. Data is streamed from disk, processed by torchmetrics, and reduced globally.
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed diagrams and design principles.
+
+---
+
+## Development
+
+This project uses modern Python tooling for dependency management and quality assurance.
+
+### Setup
+
+We use `uv` for fast dependency management.
+
+```shell
+# Install dependencies
+uv sync --all-extras
+```
+
+### Testing
+
+Tests are managed by `pytest`.
+
+```shell
+# Run all tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=evaluators
+```
+
+### Linting & Formatting
+
+We use `ruff` for all linting and formatting needs.
+
+```shell
+# Check code style
+uv run ruff check .
+
+# Format code
+uv run ruff format .
+```
+
+---
+
+## Contributing
+
+Contributions are welcome. Please ensure that:
+
+1. New features are covered by tests.
+2. Code passes all static analysis checks (`ruff`).
+3. Architecture changes are discussed in an issue first.
+
+## Acknowledgements
+
+This work was developed at the Mobile Perception Systems (MPS) lab at Eindhoven University of Technology.
+
+## License
+
+This project is licensed under the MIT License.

evaluators-2.0.0/pyproject.toml

@@ -0,0 +1,123 @@
+[build-system]
+requires = ["uv_build>=0.9.28,<0.10.0"]
+build-backend = "uv_build"
+
+[project]
+name = "evaluators"
+version = "2.0.0"
+description = "Various scene understanding and perception evaluation metrics."
+readme = "README.md"
+requires-python = ">=3.13, <3.14"
+authors = [{ name = "Kurt Stolle", email = "kurt@computer.org" }]
+classifiers = [
+  "Development Status :: 5 - Production/Stable",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Scientific/Engineering :: Image Recognition",
+  "Topic :: Software Development :: Libraries :: Python Modules",
+  "Programming Language :: Python",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Operating System :: OS Independent",
+]
+dependencies = [
+  "torch>2.7",
+  "torchvision",
+  "torchmetrics",
+  "tensordict>=0.11.0",
+  "numpy>=2.0.0",
+  "matplotlib>=3.9.0",
+  "rich>=13.9.4",
+  "lmdb",
+  "tifffile",
+  "pydantic",
+  "opencv-python>=4.13.0.92",
+]
+
+[dependency-groups]
+dev = [
+  "hypothesis>=6.140.2",
+  "nbstripout>=0.9.0",
+  "pre-commit>=4.5.1",
+  "pytest>=8.4.2",
+  "pytest-benchmark>=5.1.0",
+  "pytest-cov>=7.0.0",
+  "pytest-mock>=3.15.1",
+  "pytest-runner>=6.0.1",
+  "pytest-sugar>=1.1.1",
+  "pytest-xdist>=3.8.0",
+  "ruff>=0.9.0",
+  "ipdb>=0.13.13",
+  "ty>=0.0.17",
+  "pytest-timeout>=2.4.0",
+  "uv-build>=0.10.9",
+]
+docs = ["numpydoc>=1.9.0", "sphinx>=8.2.3", "sphinx-rtd-theme>=3.0.2"]
+notebooks = ["ipykernel>=6.30.1", "ipywidgets>=8.1.7", "jupyterlab>=4.4.9"]
+
+[tool.ruff]
+target-version = 'py313'
+line-length = 88
+
+[tool.ruff.lint]
+select = [
+  "ARG", # flake8-unused-arguments
+  "B",   # flake8-bugbear
+  "C",   # mccabe
+  "C4",  # flake8-comprehensions
+  #"DOC",  # doclint
+  "D",    # pydocstyle
+  "E",    # pycodestyle
+  "EM",   # flake8-errmsg
+  "F",    # pyflakes
+  "FURB", # refurb
+  "G",    # flake8-logging-format
+  "I",    # isort
+  "ICN",  # flake8-import-conventions
+  "N",    # pep8-naming
+  "NPY",  # NumPy specific rules
+  "PERF", # Perflint
+  "PGH",  # pygrep-hooks
+  "PIE",  # flake8-pie
+  "PL",   # pylint
+  "PT",   # flake8-pytest-style
+  "PTH",  # flake8-use-pathlib
+  "PYI",  # flake8-pyi
+  "RUF",  # Ruff-specific rules
+  "RET",  # flake8-return
+  "SIM",  # flake8-simplify
+  "T20",  # flake8-print
+  "TRY",  # Tryceratops
+  "UP",   # pyupgrade
+]
+ignore = [
+  "PLR2004", # magic numbers
+  "PLW2901", # for loop variable overwritten
+  "D212",
+  "D203",
+  "D107",    # docstring on __init__
+]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["D", "T"]
+
+[tool.ruff.format]
+docstring-code-format = true
+docstring-code-line-length = "dynamic"
+
+[tool.uv.build-backend]
+module-root = "sources"
+module-name = ["evaluators"]
+namespace = true
+
+[tool.ty.environment]
+root = ["./sources"]
+
+[tool.ty.src]
+include = ["sources", "tests", "notebooks", "scripts"]
+
+[tool.pytest.ini_options]
+timeout = 10

evaluators-2.0.0/sources/evaluators/__init__.py

@@ -0,0 +1,59 @@
+"""
+Scalable Distributed Evaluation Library.
+========================================
+
+A high-throughput evaluation framework designed for modern Computer Vision
+research. It specializes in handling large-scale video tasks (Tracking,
+STEP, DVPS) by decoupling inference I/O from metric computation.
+
+This library orchestrates a three-stage pipeline:
+
+1.  **Write:** High-speed, potentially sharded writing of inference outputs
+    to disk using memory-mapped buffers.
+2.  **Schedule:** Global indexing and load-balancing of video sequences
+    across distributed workers.
+3.  **Compute:** Lazy, streaming read access for standard metrics (HOTA, VPQ)
+    via virtual sequence views.
+
+"""
+
+from typing import TYPE_CHECKING
+
+from ._engine import run_offline_evaluation
+from ._streams import BackendInterface, MetricStream, UnboundStreamWarning
+from .memory import (
+    EncodedMemmapTemporalWriter,
+    LMDBTemporalWriter,
+    MemorySchema,
+    MemmapTemporalWriter,
+    TensorField,
+)
+
+if TYPE_CHECKING:
+    __version__: str
+    __all__ = [
+        "BackendInterface",
+        "EncodedMemmapTemporalWriter",
+        "LMDBTemporalWriter",
+        "MemorySchema",
+        "MemmapTemporalWriter",
+        "MetricStream",
+        "TensorField",
+        "UnboundStreamWarning",
+        "__version__",
+        "run_offline_evaluation",
+    ]
+
+
+# 2. Runtime Lazy Loader
+def __getattr__(name: str) -> str:
+    if name == "__version__":
+        from importlib.metadata import PackageNotFoundError, version  # noqa: PLC0415
+
+        try:
+            return version("evaluators")
+        except PackageNotFoundError:
+            return "unknown"
+
+    msg = f"module {__name__!r} has no attribute {name!r}"
+    raise AttributeError(msg)