evaluators 2.1.4__tar.gz → 3.2.0__tar.gz

evaluators-3.2.0/PKG-INFO

@@ -0,0 +1,432 @@
+Metadata-Version: 2.3
+Name: evaluators
+Version: 3.2.0
+Summary: Various scene understanding and perception evaluation metrics.
+Author: Kurt Stolle
+Author-email: Kurt Stolle <kurt@computer.org>
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Requires-Dist: torch>2.7
+Requires-Dist: torchvision
+Requires-Dist: torchmetrics
+Requires-Dist: tensordict>=0.11.0
+Requires-Dist: numpy>=2.0.0
+Requires-Dist: matplotlib>=3.9.0
+Requires-Dist: rich>=13.9.4
+Requires-Dist: pydantic
+Requires-Dist: opencv-python-headless>=4.13.0.92
+Requires-Dist: omegaconf>=2.3.0
+Requires-Dist: hydra-zen>=0.16.0
+Requires-Dist: scipy>=1.14.0
+Requires-Dist: shapely>=2.0.0
+Requires-Dist: torchmatch>=1.0.4
+Requires-Dist: blosc2>=4.1.2 ; extra == 'blosc2'
+Requires-Dist: lightning>=2.6.1 ; extra == 'lightning'
+Requires-Dist: lmdb>=2.2.0 ; extra == 'lmdb'
+Requires-Dist: imagecodecs>=2026.3.6 ; extra == 'tiff'
+Requires-Dist: tifffile>=2026.3.3 ; extra == 'tiff'
+Requires-Python: >=3.13, <3.14
+Provides-Extra: blosc2
+Provides-Extra: lightning
+Provides-Extra: lmdb
+Provides-Extra: tiff
+Description-Content-Type: text/markdown
+
+# Scalable Distributed Evaluation for Computer Vision
+
+[![PyPI](https://img.shields.io/pypi/v/evaluators)](https://pypi.org/project/evaluators/)
+[![Python Version](https://img.shields.io/pypi/pyversions/evaluators)](https://pypi.org/project/evaluators/)
+[![License](https://img.shields.io/pypi/l/evaluators)](https://github.com/khwstolle/evaluators/blob/main/LICENSE)
+[![Build Status](https://img.shields.io/github/actions/workflow/status/khwstolle/evaluators/ci.yml)](https://github.com/khwstolle/evaluators/actions)
+
+Metrics like panoptic quality over video (VPQ, DVPQ), depth-aware segmentation-and-tracking quality (DSTQ), and multi-object tracking metrics (HOTA, CLEAR, IDF1) are computed over whole sequences, not independent per-frame samples: window/tube matching across consecutive frames, or track-identity consistency across an entire video. A validation loop doesn't hand a metric a whole sequence at once. It hands it one batch at a time, and a given sequence's frames can land in different steps, different dataloader workers, and different distributed ranks depending on how the loader shards them. Standard online metric aggregation, `update()` per step, reduce across ranks at `compute()`, works when each sample's contribution reduces independently of order. It breaks down when a metric needs contiguous, ordered access to every frame of a sequence before it can produce a result.
+
+The usual workaround is to write every prediction to disk during the validation epoch and run the official reference script by hand afterward, outside the training loop. Evaluators automates that instead. Predictions still stream to a storage backend (Memory Map or LMDB) during inference, with no metric computation in the hot path. Once the epoch or run ends, the pipeline reassembles complete, correctly-ordered sequences, regardless of how their frames were scattered across steps and ranks, and computes the metric against the whole sequence automatically. Dropping `DistributedEvaluationCallback` into a PyTorch Lightning `Trainer` is the common case: whole-sequence metrics get logged like any other validation metric, with no manual step in between.
+
+Because predictions are persisted as a side effect, the same pipeline also supports deferred, offline re-evaluation: save the config during inference, then compute later, on a different machine, without re-running the model. That capability is real and useful, particularly for reproducibility, but it follows from the design; it isn't the reason the library exists.
+
+The key features are:
+
+- **Whole-sequence metrics, computed automatically.**
+  Handles metrics that standard per-step online aggregation cannot: window/tube-based video panoptic quality (VPQ, DVPQ, DSTQ) and identity-consistency tracking metrics (HOTA, CLEAR, IDF1), reassembled and computed correctly regardless of how a sequence's frames were split across steps and ranks.
+
+- **Drops into a training loop.**
+  `DistributedEvaluationCallback` binds to a PyTorch Lightning `Trainer` and handles the write, reassembly, and compute stages automatically. Predictions are also written to disk, which supports deferred, offline re-evaluation on stored results.
+
+- **Distributed by design.**  
+  Synchronizes metric computation across multiple nodes and GPUs using `torchmetrics`, `torchmatch` (Hungarian assignment for the tracking metrics), and custom scheduling logic.
+
+- **Explicit Memory Schemas.**
+  Uses Pydantic-based schemas to define data formats and encodings (PNG, TIFF, Raw) up front, ensuring type safety and storage efficiency.
+
+- **Lazy loading.**  
+  Supports referencing ground truth data from disk rather than duplicating it in memory caches, enabling evaluation of terabyte-scale datasets.
+
+- **Multi-domain.**
+  Includes verified implementations for:
+  - Segmentation: Panoptic quality (PQ), semantic mIoU.
+  - DVPS: Video panoptic quality (VPQ), depth-aware video panoptic quality (DVPQ).
+  - Depth: Eigen et al. metrics (AbsRel, RMSE).
+  - Tracking: HOTA, CLEAR (MOTA/MOTP), identity F1 (IDF1).
+  - STEP: Segmentation and tracking quality (STQ), depth-aware STQ (DSTQ).
+  - Occupancy: per-class IoU/mIoU, occupied-vs-free precision/recall/F1.
+  - 3D detection: per-class/per-threshold AP and mAP (Shapely-based 3D IoU or center-distance matching), optional TP-error decomposition and AOS.
+  - Vector maps: per-class/per-threshold AP and mAP (Chamfer or Fréchet matching), optional topology accuracy.
+  - Sensor simulation: paired image/point-cloud/radar quality, plus single-stream self-consistency (temporal, cross-modal).
+
+- **CLI.**
+  A command-line interface to inspect, index, and query saved inference results.
+
+---
+
+## Installation
+
+```shell
+pip install evaluators
+```
+
+## Quick start
+
+### In the loop (PyTorch Lightning)
+
+This is the common case: a whole-sequence metric (here `SemanticMetric`; see
+`examples/03_video_dvps.py` for the DVPQ case this pipeline exists for) is
+declared on the `LightningModule`, and `DistributedEvaluationCallback` handles
+writing, sequence reassembly, and computation automatically at the end of
+validation.
+
+```python
+import lightning as L
+from tensordict import TensorDict
+from evaluators import MemorySchema, MetricStream, TensorField
+from evaluators.integrations.lightning import DistributedEvaluationCallback
+from evaluators.metrics.domain.segmentation import SemanticKeyMap, SemanticMetric
+
+
+class SegmentationModel(L.LightningModule):
+    def __init__(self):
+        super().__init__()
+        self.layer = ...  # the actual model
+
+        keys: SemanticKeyMap = {
+            "pred": "pred_sem",
+            "target": "target_sem",
+            "sequence_id": "sequence_id",
+            "frame_index": "frame_index",
+        }
+        self.metric = SemanticMetric(key_map=keys, num_classes=19, ignore_index=255)
+
+        # Both prediction and ground truth are written to storage here; see
+        # examples/02_lazy_loading.py for fetching ground truth from the
+        # dataset instead, via `target_source`.
+        schema = MemorySchema(fields={
+            "pred_sem": TensorField(dtype="int64", shape=(1024, 2048)),
+            "target_sem": TensorField(dtype="int64", shape=(1024, 2048)),
+            "sequence_id": TensorField(dtype="int64", shape=()),
+            "frame_index": TensorField(dtype="int64", shape=()),
+        })
+        self.eval_stream = MetricStream(metrics=[self.metric], name="semantic", schema=schema)
+
+    def validation_step(self, batch, batch_idx):
+        # `batch` already provides "target_sem", "sequence_id", and
+        # "frame_index" from the dataset.
+        preds = self.layer(batch["image"]).argmax(dim=1)
+        batch["pred_sem"] = preds
+        # A leading time dimension is required even for single-frame batches.
+        self.eval_stream.update(
+            TensorDict(batch, batch_size=[preds.shape[0]]).unsqueeze(1)
+        )
+
+
+trainer = L.Trainer(callbacks=[DistributedEvaluationCallback(output_dir="./eval_cache")])
+trainer.fit(SegmentationModel(), datamodule)
+```
+
+Results are logged to the trainer's loggers like any other validation metric.
+See `examples/01_getting_started.py` for a complete, runnable version.
+
+### Manual (without Lightning)
+
+This is what `DistributedEvaluationCallback` does under the hood. Use it
+directly outside a Lightning loop, or to re-evaluate predictions that were
+already written to disk without re-running the model.
+
+The core abstractions are `MetricStream` (for writing) and `run_offline_evaluation` (for computing).
+
+#### Step 1: Inference (online)
+
+```python
+import torch
+from tensordict import TensorDict
+from evaluators import MetricStream, MemorySchema, TensorField, MemmapTemporalWriter
+from evaluators.metrics.domain.segmentation import SemanticKeyMap, SemanticMetric
+```
+
+##### 1. Configure metrics and schema
+
+```python
+### Define the source for ground truth (lazy loading)
+
+dataset = CityscapesDataset(...)
+keys: SemanticKeyMap = {
+    "pred": "pred_sem",
+    "target": "target_sem",
+    "sequence_id": "sequence_id",
+    "frame_index": "frame_index",
+}
+metric = SemanticMetric(
+    key_map=keys,
+    num_classes=19,
+    ignore_index=255,
+    target_source=dataset,
+)
+
+### Define the explicit memory schema
+### Only the prediction and metadata are written; ground truth is fetched
+### from `dataset` during the Compute stage.
+
+schema = MemorySchema(fields={
+    "pred_sem": TensorField(dtype="int64", shape=(1024, 2048)),
+    "sequence_id": TensorField(dtype="int64", shape=()),
+    "frame_index": TensorField(dtype="int64", shape=()),
+})
+```
+
+##### 2. Initialize stream and writer
+
+```python
+# Create a writer (backend)
+writer = MemmapTemporalWriter(output_dir="./inference_cache/semantic", schema=schema)
+
+# Create a stream and bind the writer
+stream = MetricStream(
+    metrics=[metric],
+    name="semantic",
+    schema=schema
+)
+stream.bind(writer)
+```
+
+##### 3. Run inference loop
+
+```python
+for batch in dataloader: # Model forward pass
+    preds = model(batch["image"])
+    batch_size = preds.shape[0]
+
+    # Push to stream (non-blocking). A TensorDict is required, with a
+    # leading time dimension even for single-frame batches.
+    stream.update(
+        TensorDict(
+            {
+                "pred_sem": preds,
+                "sequence_id": batch["sequence_id"],
+                "frame_index": batch["frame_index"],
+            },
+            batch_size=[batch_size],
+        ).unsqueeze(1)
+    )
+
+# Finalize
+writer.close()
+```
+
+    > Note: `sequence_id` and `frame_index` must be `torch.int64`.
+
+##### 4. Finalize and compute
+
+```python
+from evaluators import run_offline_evaluation
+
+# Syncs workers, builds catalog, and runs metrics
+results = run_offline_evaluation(
+    metrics=[metric],
+    artifact_dir="./inference_cache/semantic"
+)
+print(results["SemanticMetric"]["mIoU"])
+```
+
+#### Step 2: Re-evaluation (offline)
+
+Because predictions are persisted, metrics can be re-calculated or added without re-running the model.
+
+```python
+# Run evaluation on existing artifacts
+results = run_offline_evaluation(
+    metrics=[new_metric],
+    artifact_dir="./inference_cache/semantic"
+)
+```
+
+### CLI tools
+
+The library includes a CLI for managing the inference cache.
+
+_List stored sequences._
+
+```shell
+evaluators memory ls ./inference_cache
+```
+
+_Inspect specific tensor shapes._
+
+```shell
+evaluators memory inspect ./inference_cache frankfurt_000001
+```
+
+_Export to standard PyTorch file._
+
+```shell
+evaluators memory export ./inference_cache frankfurt_000001 --out my_video.pt
+```
+
+---
+
+## Supported metrics
+
+### Depth estimation
+
+Implements standard error metrics (AbsRel, SqRel, RMSE, RMSElog) and threshold accuracies ($\delta < 1.25^n$).
+
+### Segmentation
+
+- _Semantic:_ Mean intersection over union (mIoU).
+- _Panoptic:_ Panoptic quality (PQ), segmentation quality (SQ), recognition quality (RQ). Supports "Thing" and "Stuff" splits.
+
+### Depth-aware video panoptic segmentation (DVPS)
+
+Implements VPQ (video panoptic quality) and its depth-aware extension, DVPQ. Both evaluate spatio-temporal consistency using sliding window tubes over consecutive frames; DVPQ additionally gates matches by pixel-wise depth accuracy against one or more relative-error thresholds.
+
+### Tracking
+
+Multi-object tracking metrics: HOTA, CLEAR (MOTA/MOTP), and identity F1 (IDF1). Assignment (frame-level detection-to-track matching) is solved by `torchmatch` rather than `scipy.optimize.linear_sum_assignment`; see [OFFLINE_EVALUATION.md](OFFLINE_EVALUATION.md) for the migration note.
+
+### STEP
+
+Segmentation and tracking quality (STQ), and its depth-aware variant (DSTQ).
+
+### Occupancy
+
+Per-class IoU/mIoU and occupied-vs-free precision/recall/F1 over voxel-grid occupancy predictions, reusing the same statistics as `SemanticMetric`. `OccupancyMetric` (`evaluators.metrics.domain.occupancy`).
+
+### 3D detection
+
+Per-class, per-threshold average precision (AP) and mAP over 3D bounding boxes. Matching is a pluggable strategy (`IoU3DSimilarity`, backed by Shapely polygon intersection, or `CenterDistanceSimilarity`, the nuScenes convention), and curve integration is a pluggable rule (COCO's 101-point, KITTI's 40-point, or nuScenes' clipped-recall convention). Optional true-positive error decomposition (mATE, mAOE) and an orientation-aware AOS aggregate. `Box3DAveragePrecisionMetric` (`evaluators.metrics.domain.box3d`).
+
+### Vector maps
+
+Per-class, per-threshold AP and mAP over polylines (lane/vector-map elements), matched by a pluggable similarity strategy (`ChamferSimilarity` or `FrechetSimilarity`) and integrated by the same curve-integration rules as 3D detection. Optional topology accuracy over the matched polyline graph. `PolylineAveragePrecisionMetric` (`evaluators.metrics.domain.vectormap`).
+
+### Sensor simulation
+
+Paired reference-vs-simulated quality for images, point clouds, and radar (`SensorImageQualityMetric`, `PointCloudQualityMetric`, `RadarQualityMetric`), plus single-stream self-consistency metrics that need no paired reference (`TemporalConsistencyMetric`, `CrossModalAlignmentMetric`). All five live under `evaluators.metrics.domain.sensorsim`.
+
+> As with `SemanticMetric`/`PanopticQualityMetric`, none of the classes above are re-exported from the top-level `evaluators` package; import them from their domain submodule as shown.
+
+---
+
+## Architecture
+
+Reassembling whole sequences out of frames scattered across steps and ranks
+takes three stages, run automatically by `DistributedEvaluationCallback`
+during validation, or manually via the CLI.
+
+1. **Write.**  
+   Each GPU writes predictions to locally sharded files (e.g. `.memmap` or `.lmdb`). No communication occurs.
+2. **Schedule.**  
+   A synchronization barrier is reached. The main process aggregates metadata manifests from all shards to build a Global Catalog. It partitions the workload (videos, not individual frames) among workers using a greedy strategy to balance duration, so each worker gets complete, ordered sequences.
+3. **Compute.**
+   Workers iterate through their assigned Virtual Sequences. Data is streamed from disk, processed by torchmetrics, and reduced globally.
+
+See [OFFLINE_EVALUATION.md](OFFLINE_EVALUATION.md) for driving this manually via the CLI (`compute`, plus `results`, `memory`, and `visualize` for reading, inspecting, and rendering stored predictions), and detailed design principles.
+
+---
+
+## Performance
+
+`evaluators` is built for high-throughput I/O. The following benchmarks were conducted using the **Comprehensive Memory Evaluation Suite (CMES)** on a mobile workstation (i7-12700H, NVMe SSD).
+
+### Throughput (FPS)
+
+| Backend | Codec | Resolution | Write FPS | Read FPS | Compression |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| **Memmap** | Raw | 512x1024 | 250.8 | 501.6 | 1.0x |
+| **Memmap** | Blosc | 512x1024 | 59.7 | 119.3 | 0.85x |
+| **LMDB** | Raw | 512x1024 | 13.3 | 26.7 | 1.0x |
+| **LMDB** | PNG | 512x1024 | 27.2 | 13.6 | 0.62x |
+| **Filesystem**| TIFF | 512x1024 | 124.9 | 249.7 | 0.89x |
+
+### Insights
+
+- **Memmap is king for raw throughput:** The `MemmapTemporalWriter` achieves >500 FPS for read operations on mid-resolution video frames, making it ideal for fast metric computation.
+- **Blosc provides the best balance:** Using `BloscCodec` with `Memmap` offers significant storage savings with minimal CPU overhead compared to PNG/TIFF.
+- **LMDB for stability:** While slower for sequential video access, LMDB provides robust ACID compliance and is preferred for random-access metadata or small feature vectors.
+
+> Full benchmark reports, including memory usage (Peak RSS) and CPU overhead plots, are available in `docs-assets/benchmarks/` (published at `/benchmarks/` on the docs site).
+
+---
+
+## Development
+
+This project uses modern Python tooling for dependency management and quality assurance.
+
+### Setup
+
+We use `uv` for fast dependency management.
+
+```shell
+# Install dependencies
+uv sync --all-extras
+```
+
+### Testing
+
+Tests are managed by `pytest`.
+
+```shell
+# Run all tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=evaluators
+```
+
+### Linting & Formatting
+
+We use `ruff` for all linting and formatting needs.
+
+```shell
+# Check code style
+uv run ruff check .
+
+# Format code
+uv run ruff format .
+```
+
+---
+
+## Contributing
+
+Contributions are welcome. Please ensure that:
+
+1. New features are covered by tests.
+2. Code passes all static analysis checks (`ruff`).
+3. Architecture changes are discussed in an issue first.
+
+## Acknowledgements
+
+This work was developed at the Mobile Perception Systems (MPS) lab at Eindhoven University of Technology.
+
+## License
+
+This project is licensed under the MIT License.