dataflow-cv 1.6.2__tar.gz → 1.7.0__tar.gz

{dataflow_cv-1.6.2/dataflow_cv.egg-info → dataflow_cv-1.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataflow-cv
-Version: 1.6.2
+Version: 1.7.0
 Summary: A computer vision dataset processing library
 Author: DataFlow-CV Team
 License: MIT
@@ -39,7 +39,7 @@ Dynamic: license-file
 
 # DataFlow-CV
 
-> 🌊 **Everything your model doesn't do.** Convert, visualize, evaluate, and more — a single CLI for all CV data.
+> 🌊 **Everything your model doesn't do.** Analyse, convert, visualize, evaluate — a single CLI for all CV data.
 
 <p align="center">
   <a href="https://pypi.org/project/dataflow-cv/"><img src="https://img.shields.io/pypi/v/dataflow-cv.svg" alt="PyPI"></a>
@@ -55,10 +55,11 @@ Dynamic: license-file
   <img src="https://img.shields.io/badge/COCO-.json-e74c3c?style=flat-square" alt="COCO">
 </p>
 
-A computer vision dataset processing library — convert, visualize, and evaluate annotations across YOLO, LabelMe, and COCO formats.
+A computer vision dataset processing library — analyse, convert, visualize, and evaluate annotations across YOLO, LabelMe, and COCO formats.
 
 | | | |
 |:---|:---|:---|
+| 🔍 **Analyse** | Dataset statistics & train/test splitting with format auto-detection | `dataflow-cv analyse stats ...` |
 | 🔄 **Convert** | 6 directions: YOLO ↔ LabelMe ↔ COCO, plus model predictions | `dataflow-cv convert yolo2coco ...` |
 | 🎨 **Visualize** | OpenCV rendering with color-coded classes, display & save modes | `dataflow-cv visualize yolo ...` |
 | 📊 **Evaluate** | COCO mAP via pycocotools, single-threshold P/R/F1 per class | `dataflow-cv evaluate detection ...` |
@@ -88,6 +89,26 @@ cd DataFlow-CV && pip install .
 
 All required parameters (image directories, label directories, class files, output paths) are positional arguments for better usability. Use `--help` on any subcommand for detailed usage.
 
+#### 🔍 Dataset Analysis
+
+```bash
+# Dataset statistics (auto-detects YOLO / LabelMe / COCO)
+dataflow-cv analyse stats yolo_labels/ --image-dir images/ --class-file classes.txt
+dataflow-cv analyse stats labelme_json/
+dataflow-cv analyse stats coco_annotations.json
+
+# Train / test split
+dataflow-cv analyse split yolo_labels/ output/ --ratio 0.8 --seed 42 --class-file classes.txt
+dataflow-cv analyse split coco_annotations.json output/ --ratio 0.8 --seed 42
+dataflow-cv analyse split labelme_json/ output/ --ratio 0.8
+
+# Sort by count descending (default: class ID ascending)
+dataflow-cv analyse stats --sort-by count --descending yolo_labels/
+
+# Verbose logging
+dataflow-cv analyse stats --verbose yolo_labels/ --class-file classes.txt
+```
+
 #### 🔄 Format Conversion
 
 ```bash
@@ -177,6 +198,9 @@ dataflow-cv evaluate segmentation --verbose anno.json pred.json
 
 # Save results as JSON
 dataflow-cv evaluate detection --output results.json anno.json pred.json
+
+# Custom log directory
+dataflow-cv evaluate detection --verbose --log-dir logs/eval/ anno.json pred.json
 ```
 
 ##### ③ Detection vs Segmentation
@@ -192,10 +216,26 @@ Two evaluation modes, distinguished by how overlap is measured:
 
 ```python
 from dataflow.util.logging import LogConfig
+from dataflow.analyse import StatsAnalyser, SplitAnalyser
 from dataflow.convert import YoloAndCocoConverter
 from dataflow.visualize import YOLOVisualizer
 from dataflow.evaluate import DetectionEvaluator, compute_pr_f1
 
+# ── Analyse ─────────────────────────────────────────
+log_cfg = LogConfig(name="analyse", verbose=True)
+
+# Dataset statistics
+analyser = StatsAnalyser(log_config=log_cfg)
+result = analyser.analyse("yolo_labels/", class_file="classes.txt")
+print(f"{result.data.total_files} images, {result.data.total_annotations} objects")
+
+# Train/test split
+splitter = SplitAnalyser(log_config=log_cfg)
+result = splitter.analyse(
+    "yolo_labels/", "output/", ratio=0.8, seed=42, class_file="classes.txt",
+)
+print(f"Train: {result.data.train_count}, Val: {result.data.val_count}")
+
 # ── Convert ──────────────────────────────────────────
 # YOLO labels → COCO (label mode)
 log_cfg = LogConfig(name="convert", verbose=True)
@@ -270,7 +310,7 @@ For detailed developer guidance including advanced test commands, debugging, and
 
 ### 🧪 Testing
 
-**440 tests, 77% code coverage (3957 statements).**
+**471 tests, 77% code coverage (3957 statements).**
 
 ```bash
 pytest                                    # All tests
@@ -320,12 +360,13 @@ pre-commit run --all-files    # Manual run against all files
 ```
 dataflow/
 ├── label/           # Annotation handlers + data models
+├── analyse/         # Dataset statistics & train/test split
 ├── convert/         # Format converters, RLE utility, log templates
 ├── visualize/       # OpenCV-based rendering, log templates
 ├── evaluate/        # pycocotools-based metrics, log templates
 ├── util/            # Unified logging (LogManager + format helpers)
 └── cli/             # CLI entry point, commands, validation
-tests/               # Unit & integration tests (440 tests, conftest fixtures)
+tests/               # Unit & integration tests (471 tests, conftest fixtures)
 samples/             # Python API usage examples
 assets/              # Test data (det/seg by format)
 specs/               # Canonical specifications (evaluate/ + formats/ + modules/)

{dataflow_cv-1.6.2 → dataflow_cv-1.7.0}/README.md

@@ -1,6 +1,6 @@
 # DataFlow-CV
 
-> 🌊 **Everything your model doesn't do.** Convert, visualize, evaluate, and more — a single CLI for all CV data.
+> 🌊 **Everything your model doesn't do.** Analyse, convert, visualize, evaluate — a single CLI for all CV data.
 
 <p align="center">
   <a href="https://pypi.org/project/dataflow-cv/"><img src="https://img.shields.io/pypi/v/dataflow-cv.svg" alt="PyPI"></a>
@@ -16,10 +16,11 @@
   <img src="https://img.shields.io/badge/COCO-.json-e74c3c?style=flat-square" alt="COCO">
 </p>
 
-A computer vision dataset processing library — convert, visualize, and evaluate annotations across YOLO, LabelMe, and COCO formats.
+A computer vision dataset processing library — analyse, convert, visualize, and evaluate annotations across YOLO, LabelMe, and COCO formats.
 
 | | | |
 |:---|:---|:---|
+| 🔍 **Analyse** | Dataset statistics & train/test splitting with format auto-detection | `dataflow-cv analyse stats ...` |
 | 🔄 **Convert** | 6 directions: YOLO ↔ LabelMe ↔ COCO, plus model predictions | `dataflow-cv convert yolo2coco ...` |
 | 🎨 **Visualize** | OpenCV rendering with color-coded classes, display & save modes | `dataflow-cv visualize yolo ...` |
 | 📊 **Evaluate** | COCO mAP via pycocotools, single-threshold P/R/F1 per class | `dataflow-cv evaluate detection ...` |
@@ -49,6 +50,26 @@ cd DataFlow-CV && pip install .
 
 All required parameters (image directories, label directories, class files, output paths) are positional arguments for better usability. Use `--help` on any subcommand for detailed usage.
 
+#### 🔍 Dataset Analysis
+
+```bash
+# Dataset statistics (auto-detects YOLO / LabelMe / COCO)
+dataflow-cv analyse stats yolo_labels/ --image-dir images/ --class-file classes.txt
+dataflow-cv analyse stats labelme_json/
+dataflow-cv analyse stats coco_annotations.json
+
+# Train / test split
+dataflow-cv analyse split yolo_labels/ output/ --ratio 0.8 --seed 42 --class-file classes.txt
+dataflow-cv analyse split coco_annotations.json output/ --ratio 0.8 --seed 42
+dataflow-cv analyse split labelme_json/ output/ --ratio 0.8
+
+# Sort by count descending (default: class ID ascending)
+dataflow-cv analyse stats --sort-by count --descending yolo_labels/
+
+# Verbose logging
+dataflow-cv analyse stats --verbose yolo_labels/ --class-file classes.txt
+```
+
 #### 🔄 Format Conversion
 
 ```bash
@@ -138,6 +159,9 @@ dataflow-cv evaluate segmentation --verbose anno.json pred.json
 
 # Save results as JSON
 dataflow-cv evaluate detection --output results.json anno.json pred.json
+
+# Custom log directory
+dataflow-cv evaluate detection --verbose --log-dir logs/eval/ anno.json pred.json
 ```
 
 ##### ③ Detection vs Segmentation
@@ -153,10 +177,26 @@ Two evaluation modes, distinguished by how overlap is measured:
 
 ```python
 from dataflow.util.logging import LogConfig
+from dataflow.analyse import StatsAnalyser, SplitAnalyser
 from dataflow.convert import YoloAndCocoConverter
 from dataflow.visualize import YOLOVisualizer
 from dataflow.evaluate import DetectionEvaluator, compute_pr_f1
 
+# ── Analyse ─────────────────────────────────────────
+log_cfg = LogConfig(name="analyse", verbose=True)
+
+# Dataset statistics
+analyser = StatsAnalyser(log_config=log_cfg)
+result = analyser.analyse("yolo_labels/", class_file="classes.txt")
+print(f"{result.data.total_files} images, {result.data.total_annotations} objects")
+
+# Train/test split
+splitter = SplitAnalyser(log_config=log_cfg)
+result = splitter.analyse(
+    "yolo_labels/", "output/", ratio=0.8, seed=42, class_file="classes.txt",
+)
+print(f"Train: {result.data.train_count}, Val: {result.data.val_count}")
+
 # ── Convert ──────────────────────────────────────────
 # YOLO labels → COCO (label mode)
 log_cfg = LogConfig(name="convert", verbose=True)
@@ -231,7 +271,7 @@ For detailed developer guidance including advanced test commands, debugging, and
 
 ### 🧪 Testing
 
-**440 tests, 77% code coverage (3957 statements).**
+**471 tests, 77% code coverage (3957 statements).**
 
 ```bash
 pytest                                    # All tests
@@ -281,12 +321,13 @@ pre-commit run --all-files    # Manual run against all files
 ```
 dataflow/
 ├── label/           # Annotation handlers + data models
+├── analyse/         # Dataset statistics & train/test split
 ├── convert/         # Format converters, RLE utility, log templates
 ├── visualize/       # OpenCV-based rendering, log templates
 ├── evaluate/        # pycocotools-based metrics, log templates
 ├── util/            # Unified logging (LogManager + format helpers)
 └── cli/             # CLI entry point, commands, validation
-tests/               # Unit & integration tests (440 tests, conftest fixtures)
+tests/               # Unit & integration tests (471 tests, conftest fixtures)
 samples/             # Python API usage examples
 assets/              # Test data (det/seg by format)
 specs/               # Canonical specifications (evaluate/ + formats/ + modules/)

{dataflow_cv-1.6.2 → dataflow_cv-1.7.0}/dataflow/__init__.py

@@ -1,4 +1,4 @@
 """DataFlow-CV: A computer vision dataset processing library."""
 
-__version__ = "1.6.2"
+__version__ = "1.7.0"
 __author__ = "DataFlow-CV Team"

dataflow_cv-1.7.0/dataflow/analyse/__init__.py

@@ -0,0 +1,41 @@
+"""
+Analyse module — dataset introspection and preparation.
+
+Provides dataset statistics and train/test splitting for all supported
+annotation formats (YOLO, LabelMe, COCO).  Format is auto-detected from
+the label path.
+
+Key features:
+- **Dataset statistics**: Count total files, total annotations, and
+  per-class distribution
+- **Train/test split**: Deterministic dataset splitting with configurable
+  ratio and seed
+
+Example::
+
+    from dataflow.analyse import StatsAnalyser
+
+    analyser = StatsAnalyser()
+    result = analyser.analyse(
+        label_path=Path("yolo_labels/"),
+        class_file=Path("classes.txt"),
+    )
+    if result.success:
+        stats = result.data
+        print(f"Total: {stats.total_annotations} annotations")
+"""
+
+from . import utils
+from .base import AnalysisResult, BaseAnalyser, SplitResult, StatsResult
+from .split import SplitAnalyser
+from .stats import StatsAnalyser
+
+__all__ = [
+    "BaseAnalyser",
+    "AnalysisResult",
+    "StatsResult",
+    "SplitResult",
+    "StatsAnalyser",
+    "SplitAnalyser",
+    "utils",
+]

dataflow_cv-1.7.0/dataflow/analyse/base.py

@@ -0,0 +1,154 @@
+"""
+Base classes and data models for the Analyse module.
+
+Provides shared logging infrastructure (``BaseAnalyser``) and result
+containers (``AnalysisResult``, ``StatsResult``, ``SplitResult``).
+"""
+
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+
+# ---------------------------------------------------------------------------
+# Result data models
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class AnalysisResult:
+    """Top-level return type shared by all analysers.
+
+    Attributes:
+        success: Whether the analysis completed successfully.
+        data: Module-specific result data (``StatsResult`` or ``SplitResult``).
+        errors: Accumulated error messages (non-empty if ``success=False``).
+        warnings: Accumulated non-fatal warning messages.
+        log_path: Log file path (verbose mode only, ``None`` otherwise).
+    """
+
+    success: bool = True
+    data: Optional[Any] = None
+    errors: List[str] = field(default_factory=list)
+    warnings: List[str] = field(default_factory=list)
+    log_path: Optional[str] = None
+
+    def add_error(self, error: str) -> None:
+        """Add an error and mark the result as failed."""
+        self.errors.append(error)
+        self.success = False
+
+    def add_warning(self, warning: str) -> None:
+        """Add a non-fatal warning."""
+        self.warnings.append(warning)
+
+
+@dataclass
+class StatsResult:
+    """Container for dataset statistics.
+
+    Attributes:
+        total_files: Number of label files (or images for COCO).
+        total_annotations: Total annotation objects across all images.
+        per_class: ``{class_name: count}``, ordered by class_file or
+            count descending.
+        format: Detected format (``"yolo"`` | ``"labelme"`` | ``"coco"``).
+        categories: ``{class_id: class_name}`` mapping.
+    """
+
+    total_files: int
+    total_annotations: int
+    per_class: Dict[str, int]
+    format: str
+    categories: Dict[int, str] = field(default_factory=dict)
+
+
+@dataclass
+class SplitResult:
+    """Container for train/test split results.
+
+    Attributes:
+        train_count: Number of images in the training set.
+        val_count: Number of images in the validation set.
+        train_dir: Path to the train output directory.
+        val_dir: Path to the validation output directory.
+        ratio: Train ratio used (e.g., 0.8).
+        seed: Random seed used for reproducibility.
+        format: Detected format (``"yolo"`` | ``"labelme"`` | ``"coco"``).
+    """
+
+    train_count: int
+    val_count: int
+    train_dir: Path
+    val_dir: Path
+    ratio: float
+    seed: int
+    format: str
+
+
+# ---------------------------------------------------------------------------
+# Base class
+# ---------------------------------------------------------------------------
+
+
+class BaseAnalyser:
+    """Shared logging infrastructure for dataset analysers.
+
+    Concrete analysers (``StatsAnalyser``, ``SplitAnalyser``) extend this
+    class to inherit logging helpers.  Analyse operations are read-only —
+    errors are logged and accumulated in the result, never raised.
+
+    Constructor follows the project-wide ``LogConfig`` → ``LogManager``
+    pattern (see ``dataflow/evaluate/base.py``).
+    """
+
+    def __init__(self, log_config: Optional[Any] = None):
+        """Initialise the analyser.
+
+        Args:
+            log_config: Optional ``LogConfig`` instance.  If ``None``, a
+                default ``LogConfig(name="analyse")`` is used.
+        """
+        from ..util.logging import LogConfig, LogManager
+
+        if log_config is None:
+            log_config = LogConfig(name="analyse")
+        self._log_manager = LogManager(log_config)
+        self.logger = self._log_manager.logger
+
+    # ------------------------------------------------------------------
+    # Logging helpers
+    # ------------------------------------------------------------------
+
+    def _log_info(self, message: str) -> None:
+        """Log an informational message."""
+        self.logger.info(message)
+
+    def _log_warning(self, message: str) -> None:
+        """Log a warning message."""
+        self.logger.warning(message)
+
+    def _log_error(self, message: str) -> None:
+        """Log an error message.
+
+        Unlike Convert (which raises in strict mode) and Evaluate (which
+        always raises), Analyse is a read-only operation — errors are
+        accumulated in ``AnalysisResult.errors`` rather than raised.
+        """
+        self.logger.error(message)
+
+    # ------------------------------------------------------------------
+    # Result helpers
+    # ------------------------------------------------------------------
+
+    @staticmethod
+    def _create_result(data: Any = None) -> AnalysisResult:
+        """Create a new ``AnalysisResult`` with default success state.
+
+        Args:
+            data: Optional result data (``StatsResult`` or ``SplitResult``).
+
+        Returns:
+            A fresh ``AnalysisResult`` instance.
+        """
+        return AnalysisResult(success=True, data=data)

dataflow_cv-1.7.0/dataflow/analyse/log_templates.py

@@ -0,0 +1,156 @@
+"""
+Log formatting templates for the Analyse module.
+
+Pure functions that return formatted strings — they never call the
+logger directly.  The caller decides the log level and passes the
+result to ``self.logger.info()`` / ``.debug()`` / etc.
+"""
+
+from pathlib import Path
+from typing import Dict
+
+from ..util.logging import (
+    format_divider,
+    format_kv,
+    format_result_block,
+    format_section,
+    format_table,
+)
+
+
+def format_analyse_header(
+    operation: str,
+    label_path: Path,
+    format_name: str,
+) -> str:
+    """Header block shown at the start of an analysis operation.
+
+    Args:
+        operation: Operation name, e.g. ``"Dataset Statistics"``.
+        label_path: Path to the label source.
+        format_name: Detected format, e.g. ``"yolo (auto-detected)"``.
+
+    Returns:
+        Formatted header string.
+    """
+    lines = [
+        format_divider("═"),
+        f"Analyse: {operation}",
+        format_kv("Source", str(label_path)),
+        format_kv("Format", format_name),
+        "",
+    ]
+    return "\n".join(lines)
+
+
+def format_stats_result(
+    total_files: int,
+    total_annotations: int,
+    per_class: Dict[str, int],
+    categories: Dict[int, str] = None,
+) -> str:
+    """Per-class statistics table with summary.
+
+    Args:
+        total_files: Number of label files.
+        total_annotations: Total annotation objects.
+        per_class: ``{class_name: count}`` dict, pre-ordered.
+        categories: ``{class_id: class_name}`` dict for ID lookup.
+            When provided, an ``ID`` column is added (0-indexed).
+
+    Returns:
+        Formatted statistics output.
+    """
+    lines = [
+        format_section("Summary"),
+        format_kv("Total files", str(total_files)),
+        format_kv("Total annotations", str(total_annotations)),
+        format_kv("Categories", str(len(per_class))),
+        "",
+    ]
+
+    if per_class:
+        # Build name→id reverse mapping
+        name_to_id: Dict[str, int] = {}
+        if categories:
+            for cid, cname in categories.items():
+                if cname not in name_to_id:
+                    name_to_id[cname] = cid
+
+        if categories:
+            headers = ["Class", "ID", "Count"]
+            rows = [
+                [name, str(name_to_id.get(name, "")), str(count)]
+                for name, count in per_class.items()
+            ]
+            rows.append(["─" * 15, "─" * 4, "─" * 7])
+            rows.append([
+                f"Total ({len(per_class)})",
+                "",
+                str(sum(per_class.values())),
+            ])
+        else:
+            headers = ["Class", "Count"]
+            rows = [[name, str(count)] for name, count in per_class.items()]
+            rows.append(["─" * 15, "─" * 7])
+            rows.append([
+                f"Total ({len(per_class)})",
+                str(sum(per_class.values())),
+            ])
+        lines.append(format_section("Per-Class"))
+        lines.append(format_table(headers, rows))
+    else:
+        lines.append(format_section("Per-Class"))
+        lines.append("  (no annotations)")
+
+    return "\n".join(lines)
+
+
+def format_split_result(
+    train_count: int,
+    val_count: int,
+    train_dir: Path,
+    val_dir: Path,
+    ratio: float,
+    seed: int,
+) -> str:
+    """Split summary block.
+
+    Args:
+        train_count: Number of images in training set.
+        val_count: Number of images in validation set.
+        train_dir: Output directory for training data.
+        val_dir: Output directory for validation data.
+        ratio: Train ratio used.
+        seed: Random seed used.
+
+    Returns:
+        Formatted split summary.
+    """
+    total = train_count + val_count
+    lines = [
+        format_kv("Ratio", str(ratio)),
+        format_kv("Seed", str(seed)),
+        "",
+        format_section("Split"),
+        format_kv("Train", f"{train_count} images → {train_dir}"),
+        format_kv("Val", f"{val_count} images → {val_dir}"),
+        format_kv("Total", str(total)),
+    ]
+    return "\n".join(lines)
+
+
+def format_analyse_result(
+    status: str,
+    log_path: str,
+) -> str:
+    """Final result block.
+
+    Args:
+        status: Status string, e.g. ``"✓ Success"``.
+        log_path: Path to the log file (or empty string).
+
+    Returns:
+        Formatted result block.
+    """
+    return format_result_block(status, {}, log_path or None)