trace-digitiser 0.1.0__tar.gz

trace_digitiser-0.1.0/PKG-INFO

@@ -0,0 +1,176 @@
+Metadata-Version: 2.4
+Name: trace-digitiser
+Version: 0.1.0
+Summary: Template-free computer-vision digitisation pipeline for raster scientific line plots.
+Author: Trace Digitiser Contributors
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: opencv-python-headless>=4.8
+Requires-Dist: pytesseract>=0.3.10
+Requires-Dist: Pillow>=10.0
+Requires-Dist: pandas>=2.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: matplotlib>=3.7
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-cov>=4.1; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+
+# trace-digitiser
+
+Template-free computer-vision digitisation pipeline for raster scientific line plots.
+
+Given a raster image of a scientific figure (PNG, JPEG, etc.), this tool detects plot panels, reads y-axis tick labels via OCR, segments coloured traces, and exports digitised data as CSV files — all without requiring manual calibration points, known trace colours, or panel coordinates. The user supplies only a high-level layout hint (e.g. "stacked", "horizontal", "grid").
+
+## Installation
+
+Requires Python 3.10+ and [Tesseract OCR](https://github.com/tesseract-ocr/tesseract).
+
+```bash
+# Install Tesseract (Ubuntu/Debian)
+sudo apt-get install tesseract-ocr
+
+# Install Tesseract (macOS)
+brew install tesseract
+
+# Install the package
+pip install -e .
+
+# With development dependencies
+pip install -e ".[dev]"
+```
+
+## Quick start
+
+### Python API
+
+```python
+from trace_digitiser import digitise
+
+result = digitise(
+    "figure.jpg",
+    layout_mode="stacked",
+    expected_rows=2,
+    expected_cols=1,
+    output_dir="outputs",
+)
+
+# Digitised traces as a DataFrame
+print(result.trace_data.head())
+
+# Panel metadata
+for panel in result.panels:
+    print(f"Panel {panel.panel_id}: {panel.width}×{panel.height}px, "
+          f"calibration={panel.calibration.scale_type}")
+```
+
+### Command line
+
+```bash
+# Single image
+trace-digitiser figure.jpg --layout stacked --rows 2 --cols 1 -o outputs/
+
+# Batch processing
+trace-digitiser figures/*.jpg --layout auto -o results/
+
+# Generate synthetic test figures
+trace-digitiser --generate-test-figures -o test_figures/
+```
+
+### Layout modes
+
+| Mode | Use case | Example |
+|------|----------|---------|
+| `single` | One plot panel | `--layout single` |
+| `stacked` | Vertically stacked panels | `--layout stacked --rows 2 --cols 1` |
+| `horizontal` | Side-by-side panels | `--layout horizontal --rows 1 --cols 3` |
+| `grid` | Row×column subplot grid | `--layout grid --rows 2 --cols 2` |
+| `auto` | Unconstrained detection | `--layout auto` |
+
+## Output files
+
+For input `figure.jpg` with default settings:
+
+| File | Description |
+|------|-------------|
+| `figure_automated_digitised_trace_by_column.csv` | One row per x-pixel column per trace, with `center_estimate`, `upper_envelope`, `lower_envelope` |
+| `figure_automated_panel_metadata.csv` | Panel coordinates, calibration parameters, detected x-labels |
+| `figure_automated_digitised_summary_by_label.csv` | Summary statistics per detected x-label interval (written only if labels are found) |
+
+## Project structure
+
+```
+trace_digitiser/
+├── pyproject.toml
+├── src/trace_digitiser/
+│   ├── __init__.py          # Public API: digitise()
+│   ├── models.py            # Dataclasses: Panel, Calibration, Trace, DigitiserResult
+│   ├── io.py                # Image loading, CSV output
+│   ├── geometry.py          # Box area/IoU/containment helpers
+│   ├── line_detection.py    # Horizontal and vertical line detectors
+│   ├── panel_detection.py   # Candidate generation, deduplication, layout selection
+│   ├── calibration.py       # Y-axis OCR calibration (linear + log) and cross-row propagation
+│   ├── x_calibration.py     # X-axis OCR calibration (numeric ticks)
+│   ├── ocr.py               # Tesseract OCR wrappers for tick and x-label reading
+│   ├── trace_detection.py   # HSV/CIELAB colour segmentation + achromatic trace detection
+│   ├── digitise.py          # Column-by-column trace digitisation
+│   ├── summarise.py         # Interval-label summarisation
+│   ├── diagnostics.py       # Debug overlays and diagnostic file output
+│   ├── synthetic.py         # Synthetic test-figure generation
+│   └── cli.py               # Command-line interface
+└── tests/
+    ├── conftest.py           # Shared fixtures (synthetic images)
+    ├── test_geometry.py
+    ├── test_panel_detection.py
+    ├── test_trace_detection.py
+    └── test_integration.py   # Panel count, calibration quality, trace RMSE
+```
+
+## Development
+
+```bash
+# Run tests
+pytest
+
+# Run tests with coverage
+pytest --cov=trace_digitiser
+
+# Lint
+ruff check src/ tests/
+```
+
+## Limitations
+
+This tool digitises visible pixels from raster images. It does not recover original raw data. Key limitations:
+
+- **Linear and log y-axes supported** — broken, dual, and other nonlinear axes are not supported yet.
+- **X-axis calibration is best-effort** — numeric x-ticks are OCR'd when possible; otherwise x-values are normalised 0–1 or labelled by interval.
+- **Black/grey traces partially supported** — achromatic trace detection is available but works best when traces have enough contrast against the background.
+- **Requires visible structure** — axes, borders, or gridlines must be present for panel detection.
+- **OCR fragility** — small, rotated, or low-contrast tick labels may fail.
+
+## Changelog
+
+### v0.1.0
+
+**Refactored from notebook to installable package** with 15 modules, dataclasses, CLI, and test suite.
+
+**Panel detection improvements:**
+- Hint-aware `split_y_clusters` — uses the user's `expected_rows` to split evenly-spaced gridlines at the largest gaps, even when the inter-panel gap is only marginally wider than intra-panel gaps.
+- Post-hoc panel subdivision — `apply_layout_hint` can split oversized candidates at their gridlines when initial detection yields fewer panels than expected.
+- Improved grid detection — properly cross-matches row/column structure.
+
+**Trace detection improvements:**
+- CIELAB clustering mode (`use_lab=True`) — clusters non-background pixels in perceptually uniform colour space using mini-batch k-means.
+- Achromatic trace detection — a separate pass detects black/grey traces by looking for low-saturation, horizontally continuous structures distinct from grid/axis lines.
+
+**Calibration improvements:**
+- Log-scale y-axis detection — when OCR'd tick values are better explained by `log10(y_value) = a * y_pixel + b`, the calibration uses log scale automatically.
+- X-axis calibration — OCRs numeric x-axis ticks, fits `x_value = a * x_pixel + b`, and adds `x_value` column to the trace CSV.
+
+**Infrastructure:**
+- Diagnostic overlays save to files via `save_diagnostics=True`.
+- Tesseract error handling — graceful recovery from OCR crashes on degenerate crops.
+- No Colab dependency.
+- Quantitative test suite with panel count, calibration quality, and trace RMSE metrics against synthetic ground truth.

trace_digitiser-0.1.0/README.md

@@ -0,0 +1,157 @@
+# trace-digitiser
+
+Template-free computer-vision digitisation pipeline for raster scientific line plots.
+
+Given a raster image of a scientific figure (PNG, JPEG, etc.), this tool detects plot panels, reads y-axis tick labels via OCR, segments coloured traces, and exports digitised data as CSV files — all without requiring manual calibration points, known trace colours, or panel coordinates. The user supplies only a high-level layout hint (e.g. "stacked", "horizontal", "grid").
+
+## Installation
+
+Requires Python 3.10+ and [Tesseract OCR](https://github.com/tesseract-ocr/tesseract).
+
+```bash
+# Install Tesseract (Ubuntu/Debian)
+sudo apt-get install tesseract-ocr
+
+# Install Tesseract (macOS)
+brew install tesseract
+
+# Install the package
+pip install -e .
+
+# With development dependencies
+pip install -e ".[dev]"
+```
+
+## Quick start
+
+### Python API
+
+```python
+from trace_digitiser import digitise
+
+result = digitise(
+    "figure.jpg",
+    layout_mode="stacked",
+    expected_rows=2,
+    expected_cols=1,
+    output_dir="outputs",
+)
+
+# Digitised traces as a DataFrame
+print(result.trace_data.head())
+
+# Panel metadata
+for panel in result.panels:
+    print(f"Panel {panel.panel_id}: {panel.width}×{panel.height}px, "
+          f"calibration={panel.calibration.scale_type}")
+```
+
+### Command line
+
+```bash
+# Single image
+trace-digitiser figure.jpg --layout stacked --rows 2 --cols 1 -o outputs/
+
+# Batch processing
+trace-digitiser figures/*.jpg --layout auto -o results/
+
+# Generate synthetic test figures
+trace-digitiser --generate-test-figures -o test_figures/
+```
+
+### Layout modes
+
+| Mode | Use case | Example |
+|------|----------|---------|
+| `single` | One plot panel | `--layout single` |
+| `stacked` | Vertically stacked panels | `--layout stacked --rows 2 --cols 1` |
+| `horizontal` | Side-by-side panels | `--layout horizontal --rows 1 --cols 3` |
+| `grid` | Row×column subplot grid | `--layout grid --rows 2 --cols 2` |
+| `auto` | Unconstrained detection | `--layout auto` |
+
+## Output files
+
+For input `figure.jpg` with default settings:
+
+| File | Description |
+|------|-------------|
+| `figure_automated_digitised_trace_by_column.csv` | One row per x-pixel column per trace, with `center_estimate`, `upper_envelope`, `lower_envelope` |
+| `figure_automated_panel_metadata.csv` | Panel coordinates, calibration parameters, detected x-labels |
+| `figure_automated_digitised_summary_by_label.csv` | Summary statistics per detected x-label interval (written only if labels are found) |
+
+## Project structure
+
+```
+trace_digitiser/
+├── pyproject.toml
+├── src/trace_digitiser/
+│   ├── __init__.py          # Public API: digitise()
+│   ├── models.py            # Dataclasses: Panel, Calibration, Trace, DigitiserResult
+│   ├── io.py                # Image loading, CSV output
+│   ├── geometry.py          # Box area/IoU/containment helpers
+│   ├── line_detection.py    # Horizontal and vertical line detectors
+│   ├── panel_detection.py   # Candidate generation, deduplication, layout selection
+│   ├── calibration.py       # Y-axis OCR calibration (linear + log) and cross-row propagation
+│   ├── x_calibration.py     # X-axis OCR calibration (numeric ticks)
+│   ├── ocr.py               # Tesseract OCR wrappers for tick and x-label reading
+│   ├── trace_detection.py   # HSV/CIELAB colour segmentation + achromatic trace detection
+│   ├── digitise.py          # Column-by-column trace digitisation
+│   ├── summarise.py         # Interval-label summarisation
+│   ├── diagnostics.py       # Debug overlays and diagnostic file output
+│   ├── synthetic.py         # Synthetic test-figure generation
+│   └── cli.py               # Command-line interface
+└── tests/
+    ├── conftest.py           # Shared fixtures (synthetic images)
+    ├── test_geometry.py
+    ├── test_panel_detection.py
+    ├── test_trace_detection.py
+    └── test_integration.py   # Panel count, calibration quality, trace RMSE
+```
+
+## Development
+
+```bash
+# Run tests
+pytest
+
+# Run tests with coverage
+pytest --cov=trace_digitiser
+
+# Lint
+ruff check src/ tests/
+```
+
+## Limitations
+
+This tool digitises visible pixels from raster images. It does not recover original raw data. Key limitations:
+
+- **Linear and log y-axes supported** — broken, dual, and other nonlinear axes are not supported yet.
+- **X-axis calibration is best-effort** — numeric x-ticks are OCR'd when possible; otherwise x-values are normalised 0–1 or labelled by interval.
+- **Black/grey traces partially supported** — achromatic trace detection is available but works best when traces have enough contrast against the background.
+- **Requires visible structure** — axes, borders, or gridlines must be present for panel detection.
+- **OCR fragility** — small, rotated, or low-contrast tick labels may fail.
+
+## Changelog
+
+### v0.1.0
+
+**Refactored from notebook to installable package** with 15 modules, dataclasses, CLI, and test suite.
+
+**Panel detection improvements:**
+- Hint-aware `split_y_clusters` — uses the user's `expected_rows` to split evenly-spaced gridlines at the largest gaps, even when the inter-panel gap is only marginally wider than intra-panel gaps.
+- Post-hoc panel subdivision — `apply_layout_hint` can split oversized candidates at their gridlines when initial detection yields fewer panels than expected.
+- Improved grid detection — properly cross-matches row/column structure.
+
+**Trace detection improvements:**
+- CIELAB clustering mode (`use_lab=True`) — clusters non-background pixels in perceptually uniform colour space using mini-batch k-means.
+- Achromatic trace detection — a separate pass detects black/grey traces by looking for low-saturation, horizontally continuous structures distinct from grid/axis lines.
+
+**Calibration improvements:**
+- Log-scale y-axis detection — when OCR'd tick values are better explained by `log10(y_value) = a * y_pixel + b`, the calibration uses log scale automatically.
+- X-axis calibration — OCRs numeric x-axis ticks, fits `x_value = a * x_pixel + b`, and adds `x_value` column to the trace CSV.
+
+**Infrastructure:**
+- Diagnostic overlays save to files via `save_diagnostics=True`.
+- Tesseract error handling — graceful recovery from OCR crashes on degenerate crops.
+- No Colab dependency.
+- Quantitative test suite with panel count, calibration quality, and trace RMSE metrics against synthetic ground truth.

trace_digitiser-0.1.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "trace-digitiser"
+version = "0.1.0"
+description = "Template-free computer-vision digitisation pipeline for raster scientific line plots."
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+authors = [{ name = "Trace Digitiser Contributors" }]
+
+dependencies = [
+    "opencv-python-headless>=4.8",
+    "pytesseract>=0.3.10",
+    "Pillow>=10.0",
+    "pandas>=2.0",
+    "numpy>=1.24",
+    "matplotlib>=3.7",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.4",
+    "pytest-cov>=4.1",
+    "ruff>=0.4",
+]
+
+[project.scripts]
+trace-digitiser = "trace_digitiser.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-v --tb=short"
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"

trace_digitiser-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

trace_digitiser-0.1.0/src/trace_digitiser/__init__.py

@@ -0,0 +1,222 @@
+"""trace_digitiser — template-free digitisation of raster scientific line plots.
+
+Quick start::
+
+    from trace_digitiser import digitise
+
+    result = digitise(
+        "figure.jpg",
+        layout_mode="stacked",
+        expected_rows=2,
+        expected_cols=1,
+        output_dir="outputs",
+    )
+
+    print(result.trace_data.head())
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Optional
+
+import pandas as pd
+
+from .calibration import propagate_y_calibration_across_rows, robust_y_calibration
+from .diagnostics import draw_digitised_trace, draw_panel_overlay, draw_trace_mask, print_panel_summary
+from .digitise import digitise_trace_mask
+from .io import build_panel_metadata, load_image, save_outputs
+from .models import Calibration, DigitiserResult, Panel, Trace, XLabel
+from .ocr import detect_x_labels
+from .panel_detection import find_plot_panels
+from .summarise import summarise_by_detected_labels
+from .trace_detection import detect_trace_masks
+from .x_calibration import calibrate_x_axis
+
+__all__ = [
+    "digitise",
+    "Calibration",
+    "DigitiserResult",
+    "Panel",
+    "Trace",
+    "XLabel",
+]
+
+
+def digitise(
+    image_path: str | Path,
+    *,
+    layout_mode: str = "auto",
+    expected_rows: Optional[int] = None,
+    expected_cols: Optional[int] = None,
+    expected_panels: Optional[int] = None,
+    output_dir: Optional[str | Path] = None,
+    output_prefix: Optional[str] = None,
+    show_debug: bool = False,
+    save_diagnostics: bool = False,
+) -> DigitiserResult:
+    """End-to-end chart digitisation pipeline.
+
+    Parameters
+    ----------
+    image_path : str or Path
+        Path to the input raster image.
+    layout_mode : str
+        ``"auto"``, ``"single"``, ``"stacked"``, ``"horizontal"``, or
+        ``"grid"``.
+    expected_rows, expected_cols, expected_panels : int, optional
+        Layout hints that constrain panel selection.
+    output_dir : str or Path, optional
+        Directory for CSV outputs.  Defaults to current directory.
+    output_prefix : str, optional
+        Prefix for output filenames.  Defaults to the image stem.
+    show_debug : bool
+        If True, display inline diagnostic plots (for interactive use).
+    save_diagnostics : bool
+        If True, write diagnostic overlay PNGs to *output_dir*.
+
+    Returns
+    -------
+    DigitiserResult
+        Structured result with panels, traces, DataFrames, and paths.
+    """
+    image_path = Path(image_path)
+    rgb = load_image(image_path)
+
+    if output_prefix is None:
+        output_prefix = image_path.stem
+
+    diag_dir: Optional[Path] = None
+    if save_diagnostics:
+        diag_dir = Path(output_dir or ".") / "diagnostics"
+        diag_dir.mkdir(parents=True, exist_ok=True)
+
+    if show_debug:
+        print("Processing:", image_path)
+        print("Image size:", rgb.shape[1], "×", rgb.shape[0])
+
+    # ------------------------------------------------------------------
+    # 1. Detect panels
+    # ------------------------------------------------------------------
+    panels, h_lines, v_lines = find_plot_panels(
+        rgb,
+        layout_mode=layout_mode,
+        expected_rows=expected_rows,
+        expected_cols=expected_cols,
+        expected_panels=expected_panels,
+    )
+
+    if show_debug or save_diagnostics:
+        print_panel_summary(panels, h_lines, v_lines, layout_mode)
+        draw_panel_overlay(rgb, panels, h_lines, v_lines, output_dir=diag_dir, show=show_debug)
+
+    # ------------------------------------------------------------------
+    # 2. Y-axis calibration
+    # ------------------------------------------------------------------
+    for p in panels:
+        calib = robust_y_calibration(rgb, p, verbose=show_debug)
+        p["y_calibration"] = calib.to_dict()
+
+    panels = propagate_y_calibration_across_rows(panels, verbose=show_debug)
+
+    # ------------------------------------------------------------------
+    # 2b. X-axis calibration (numeric x ticks)
+    # ------------------------------------------------------------------
+    for p in panels:
+        x_cal = calibrate_x_axis(rgb, p, verbose=show_debug)
+        if x_cal is not None:
+            p["x_calibration"] = x_cal
+
+    # ------------------------------------------------------------------
+    # 3. Trace detection and digitisation
+    # ------------------------------------------------------------------
+    all_trace_frames: list[pd.DataFrame] = []
+    trace_debug: list[tuple[dict, dict]] = []
+
+    for p in panels:
+        masks = detect_trace_masks(rgb, p)
+        if show_debug:
+            print(f"Panel {p['panel_id']}: detected {len(masks)} coloured trace(s)")
+
+        for tr in masks:
+            if show_debug:
+                print(" ", {k: v for k, v in tr.items() if k != "mask"})
+            all_trace_frames.append(digitise_trace_mask(p, tr))
+            trace_debug.append((p, tr))
+
+    trace_data = pd.concat(all_trace_frames, ignore_index=True) if all_trace_frames else pd.DataFrame()
+
+    # ------------------------------------------------------------------
+    # 4. X-label OCR
+    # ------------------------------------------------------------------
+    for p in panels:
+        p["x_labels"] = detect_x_labels(rgb, p)
+        if show_debug:
+            print(f"Panel {p['panel_id']} x labels:")
+            for lab in p["x_labels"]:
+                print(f"  {lab['text']:>8s}  x={lab['x']:.1f}  conf={lab['conf']:.1f}")
+
+    # ------------------------------------------------------------------
+    # 5. Interval summaries
+    # ------------------------------------------------------------------
+    summary_by_label = summarise_by_detected_labels(trace_data, panels)
+
+    # ------------------------------------------------------------------
+    # 6. Diagnostics
+    # ------------------------------------------------------------------
+    if show_debug or save_diagnostics:
+        for p, tr in trace_debug:
+            draw_trace_mask(rgb, p, tr, output_dir=diag_dir, show=show_debug)
+
+        if not trace_data.empty:
+            for (panel_id, trace_id), _ in trace_data.groupby(["panel_id", "trace_id"]):
+                draw_digitised_trace(trace_data, panel_id, trace_id, output_dir=diag_dir, show=show_debug)
+
+    # ------------------------------------------------------------------
+    # 7. Save outputs
+    # ------------------------------------------------------------------
+    panel_metadata = build_panel_metadata(panels)
+    trace_csv, meta_csv, summary_csv = save_outputs(
+        trace_data, panel_metadata, summary_by_label, output_prefix, output_dir
+    )
+
+    if show_debug:
+        print("Wrote:")
+        print(" -", trace_csv)
+        print(" -", meta_csv)
+        if summary_csv:
+            print(" -", summary_csv)
+        else:
+            print(" - no label summary; fewer than two labels detected")
+
+    # ------------------------------------------------------------------
+    # 8. Build structured result
+    # ------------------------------------------------------------------
+    return DigitiserResult(
+        image_path=image_path,
+        rgb=rgb,
+        panels=[
+            Panel(
+                panel_id=p["panel_id"],
+                x0=p["x0"],
+                x1=p["x1"],
+                y_top=p["y_top"],
+                y_bottom=p["y_bottom"],
+                gridline_y=p["gridline_y"],
+                source=p["source"],
+                score=p["score"],
+                layout_mode=p.get("layout_mode", layout_mode),
+                calibration=Calibration(**p["y_calibration"]) if "y_calibration" in p else None,
+                x_labels=[
+                    XLabel(**lab) for lab in p.get("x_labels", [])
+                ],
+            )
+            for p in panels
+        ],
+        trace_data=trace_data,
+        summary_by_label=summary_by_label,
+        panel_metadata=panel_metadata,
+        trace_csv_path=trace_csv,
+        summary_csv_path=summary_csv,
+        metadata_csv_path=meta_csv,
+    )