ocr-bench 0.0.1__tar.gz

ocr_bench-0.0.1/PKG-INFO

@@ -0,0 +1,131 @@
+Metadata-Version: 2.4
+Name: ocr-bench
+Version: 0.0.1
+Summary: OCR model evaluation toolkit — VLM-as-judge with per-dataset leaderboards
+Author: Daniel van Strien
+Author-email: Daniel van Strien <davanstrien@gmail.com>
+License-Expression: Apache-2.0
+Requires-Dist: datasets>=4.0.0
+Requires-Dist: huggingface-hub
+Requires-Dist: numpy
+Requires-Dist: openai
+Requires-Dist: pillow
+Requires-Dist: rich
+Requires-Dist: scipy
+Requires-Dist: stamina
+Requires-Dist: structlog
+Requires-Dist: tqdm
+Requires-Dist: fastapi>=0.115 ; extra == 'viewer'
+Requires-Dist: uvicorn[standard]>=0.32 ; extra == 'viewer'
+Requires-Dist: jinja2>=3.1 ; extra == 'viewer'
+Requires-Dist: python-multipart>=0.0.9 ; extra == 'viewer'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://github.com/davanstrien/ocr-bench
+Project-URL: Repository, https://github.com/davanstrien/ocr-bench
+Project-URL: Demo, https://huggingface.co/spaces/davanstrien/ocr-bench-britannica-results-qwen35-viewer
+Provides-Extra: viewer
+Description-Content-Type: text/markdown
+
+# ocr-bench
+
+**There is no single best OCR model.** Rankings change depending on your documents — manuscript cards, printed books, and historical texts all produce different winners.
+
+ocr-bench allows you to create **per-collection leaderboards** using a VLM-as-judge approach, so you can find what works best for _your_ documents rather than relying on generic benchmarks. You can validate the VLM's judgement with human votes, and share results via the Hugging Face Hub.
+
+## Why?
+
+Generic OCR benchmarks tell you which model wins _on average_. But if you're digitising 18th-century encyclopaedias, that average doesn't help — the best model for your documents might be the worst on someone else's.
+
+ocr-bench lets you run the same set of OCR models on a sample of _your_ collection, then uses a vision-language model to judge which produces the best transcription for each document. The result is a leaderboard specific to your data.
+
+| Model              | BPL card catalog | Britannica 1771 |
+| ------------------ | :--------------: | :-------------: |
+| GLM-OCR (0.9B)     |    #2 (1535)     |  **#1** (1787)  |
+| LightOnOCR-2 (1B)  |  **#1** (1559)   |    #2 (1780)    |
+| FireRed-OCR (2.1B) |        —         |    #3 (1551)    |
+| DeepSeek-OCR (4B)  |    #4 (1452)     |    #4 (1437)    |
+| dots.ocr (1.7B)    |    #3 (1453)     |    #5 (945)     |
+
+Rankings flip completely between collections. The model that's #1 on BPL cards drops to #2 on Britannica, and dots.ocr goes from #3 to dead last.
+
+## Hub-native by design
+
+The entire evaluation loop lives on the Hugging Face Hub:
+
+1. **Your dataset** on the Hub (images + optional ground truth)
+2. **OCR models** run via [HF Jobs](https://huggingface.co/docs/hub/jobs-overview) → outputs written as PRs on a Hub dataset
+3. **VLM judge** via [HF Inference Providers](https://huggingface.co/docs/inference-providers/index) — only needs an HF token
+4. **Results** published to a Hub dataset (leaderboard + pairwise comparisons)
+5. **Viewer** as a [HF Space](https://huggingface.co/spaces) for browsing and human validation
+
+No local GPU required. Everything is shareable via Hub URLs.
+
+## Quickstart
+
+```bash
+pip install ocr-bench[viewer]
+
+# 1. Run OCR models on your dataset
+ocr-bench run <input-dataset> <output-repo> --max-samples 50
+
+# 2. Judge outputs pairwise with a VLM
+ocr-bench judge <output-repo>
+
+# 3. Browse results + validate
+ocr-bench view <output-repo>-results
+```
+
+## How it works
+
+**`ocr-bench run`** launches OCR models on your dataset via [HF Jobs](https://huggingface.co/docs/hub/jobs-overview). Each model writes its output as a PR on the same Hub dataset, keeping everything together without merge conflicts.
+
+**`ocr-bench judge`** runs pairwise comparisons using a VLM judge (default: [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) via HF Inference Providers). For each document, the judge sees the original image and two OCR outputs (anonymised as A/B) and picks the better transcription. Results are fit to a [Bradley-Terry model](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model) to produce ELO ratings with bootstrap 95% confidence intervals. Adaptive stopping halts early when rankings are statistically resolved.
+
+**`ocr-bench view`** serves a local web viewer with a leaderboard, comparison browser, and human validation. Vote on comparisons to cross-check the automated judge with human judgement.
+
+## Available models
+
+ocr-bench ships with 5 OCR models ready to run:
+
+| Model | Size | Best for | Notes |
+|-------|------|----------|-------|
+| `glm-ocr` | 0.9B | Historical printed text | Top performer on Britannica |
+| `lighton-ocr-2` | 1B | Card catalogs, manuscripts | Top performer on BPL |
+| `firered-ocr` | 2.1B | Clean printed text | Mid-pack on degraded docs |
+| `deepseek-ocr` | 4B | Diverse documents | Most consistent across types |
+| `dots-ocr` | 1.7B | General | Struggles on historical text |
+
+All model scripts are available at [uv-scripts/ocr](https://huggingface.co/datasets/uv-scripts/ocr) on the Hub.
+
+By default all 5 run. To pick specific models:
+
+```bash
+ocr-bench run <dataset> <output> --models glm-ocr lighton-ocr-2
+```
+
+## Example results
+
+Browse these on the Hub:
+
+- [davanstrien/ocr-bench-britannica-results-qwen35](https://huggingface.co/datasets/davanstrien/ocr-bench-britannica-results-qwen35) — Encyclopaedia Britannica 1771, 5 models, 50 samples
+- [davanstrien/bpl-ocr-bench-results](https://huggingface.co/datasets/davanstrien/bpl-ocr-bench-results) — Boston Public Library card catalog, 4 models, 150 samples
+- [Live viewer](https://huggingface.co/spaces/davanstrien/ocr-bench-britannica-results-qwen35-viewer) — Britannica leaderboard with ELO chart and comparison browser
+
+## Install
+
+```bash
+pip install ocr-bench            # Core (run + judge)
+pip install ocr-bench[viewer]    # With web UI
+```
+
+Or with [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv pip install ocr-bench[viewer]
+```
+
+Requires Python >= 3.11 and an [HF token](https://huggingface.co/settings/tokens).
+
+## Status
+
+Working proof of concept. The core pipeline (run → judge → view) is functional. Not polished production software — expect rough edges.

ocr_bench-0.0.1/README.md

@@ -0,0 +1,103 @@
+# ocr-bench
+
+**There is no single best OCR model.** Rankings change depending on your documents — manuscript cards, printed books, and historical texts all produce different winners.
+
+ocr-bench allows you to create **per-collection leaderboards** using a VLM-as-judge approach, so you can find what works best for _your_ documents rather than relying on generic benchmarks. You can validate the VLM's judgement with human votes, and share results via the Hugging Face Hub.
+
+## Why?
+
+Generic OCR benchmarks tell you which model wins _on average_. But if you're digitising 18th-century encyclopaedias, that average doesn't help — the best model for your documents might be the worst on someone else's.
+
+ocr-bench lets you run the same set of OCR models on a sample of _your_ collection, then uses a vision-language model to judge which produces the best transcription for each document. The result is a leaderboard specific to your data.
+
+| Model              | BPL card catalog | Britannica 1771 |
+| ------------------ | :--------------: | :-------------: |
+| GLM-OCR (0.9B)     |    #2 (1535)     |  **#1** (1787)  |
+| LightOnOCR-2 (1B)  |  **#1** (1559)   |    #2 (1780)    |
+| FireRed-OCR (2.1B) |        —         |    #3 (1551)    |
+| DeepSeek-OCR (4B)  |    #4 (1452)     |    #4 (1437)    |
+| dots.ocr (1.7B)    |    #3 (1453)     |    #5 (945)     |
+
+Rankings flip completely between collections. The model that's #1 on BPL cards drops to #2 on Britannica, and dots.ocr goes from #3 to dead last.
+
+## Hub-native by design
+
+The entire evaluation loop lives on the Hugging Face Hub:
+
+1. **Your dataset** on the Hub (images + optional ground truth)
+2. **OCR models** run via [HF Jobs](https://huggingface.co/docs/hub/jobs-overview) → outputs written as PRs on a Hub dataset
+3. **VLM judge** via [HF Inference Providers](https://huggingface.co/docs/inference-providers/index) — only needs an HF token
+4. **Results** published to a Hub dataset (leaderboard + pairwise comparisons)
+5. **Viewer** as a [HF Space](https://huggingface.co/spaces) for browsing and human validation
+
+No local GPU required. Everything is shareable via Hub URLs.
+
+## Quickstart
+
+```bash
+pip install ocr-bench[viewer]
+
+# 1. Run OCR models on your dataset
+ocr-bench run <input-dataset> <output-repo> --max-samples 50
+
+# 2. Judge outputs pairwise with a VLM
+ocr-bench judge <output-repo>
+
+# 3. Browse results + validate
+ocr-bench view <output-repo>-results
+```
+
+## How it works
+
+**`ocr-bench run`** launches OCR models on your dataset via [HF Jobs](https://huggingface.co/docs/hub/jobs-overview). Each model writes its output as a PR on the same Hub dataset, keeping everything together without merge conflicts.
+
+**`ocr-bench judge`** runs pairwise comparisons using a VLM judge (default: [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) via HF Inference Providers). For each document, the judge sees the original image and two OCR outputs (anonymised as A/B) and picks the better transcription. Results are fit to a [Bradley-Terry model](https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model) to produce ELO ratings with bootstrap 95% confidence intervals. Adaptive stopping halts early when rankings are statistically resolved.
+
+**`ocr-bench view`** serves a local web viewer with a leaderboard, comparison browser, and human validation. Vote on comparisons to cross-check the automated judge with human judgement.
+
+## Available models
+
+ocr-bench ships with 5 OCR models ready to run:
+
+| Model | Size | Best for | Notes |
+|-------|------|----------|-------|
+| `glm-ocr` | 0.9B | Historical printed text | Top performer on Britannica |
+| `lighton-ocr-2` | 1B | Card catalogs, manuscripts | Top performer on BPL |
+| `firered-ocr` | 2.1B | Clean printed text | Mid-pack on degraded docs |
+| `deepseek-ocr` | 4B | Diverse documents | Most consistent across types |
+| `dots-ocr` | 1.7B | General | Struggles on historical text |
+
+All model scripts are available at [uv-scripts/ocr](https://huggingface.co/datasets/uv-scripts/ocr) on the Hub.
+
+By default all 5 run. To pick specific models:
+
+```bash
+ocr-bench run <dataset> <output> --models glm-ocr lighton-ocr-2
+```
+
+## Example results
+
+Browse these on the Hub:
+
+- [davanstrien/ocr-bench-britannica-results-qwen35](https://huggingface.co/datasets/davanstrien/ocr-bench-britannica-results-qwen35) — Encyclopaedia Britannica 1771, 5 models, 50 samples
+- [davanstrien/bpl-ocr-bench-results](https://huggingface.co/datasets/davanstrien/bpl-ocr-bench-results) — Boston Public Library card catalog, 4 models, 150 samples
+- [Live viewer](https://huggingface.co/spaces/davanstrien/ocr-bench-britannica-results-qwen35-viewer) — Britannica leaderboard with ELO chart and comparison browser
+
+## Install
+
+```bash
+pip install ocr-bench            # Core (run + judge)
+pip install ocr-bench[viewer]    # With web UI
+```
+
+Or with [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv pip install ocr-bench[viewer]
+```
+
+Requires Python >= 3.11 and an [HF token](https://huggingface.co/settings/tokens).
+
+## Status
+
+Working proof of concept. The core pipeline (run → judge → view) is functional. Not polished production software — expect rough edges.

ocr_bench-0.0.1/pyproject.toml

@@ -0,0 +1,50 @@
+[project]
+name = "ocr-bench"
+version = "0.0.1"
+description = "OCR model evaluation toolkit — VLM-as-judge with per-dataset leaderboards"
+readme = "README.md"
+authors = [
+    { name = "Daniel van Strien", email = "davanstrien@gmail.com" }
+]
+requires-python = ">=3.11"
+license = "Apache-2.0"
+dependencies = [
+    "datasets>=4.0.0",
+    "huggingface-hub",
+    "numpy",
+    "openai",
+    "pillow",
+    "rich",
+    "scipy",
+    "stamina",
+    "structlog",
+    "tqdm",
+]
+
+[project.urls]
+Homepage = "https://github.com/davanstrien/ocr-bench"
+Repository = "https://github.com/davanstrien/ocr-bench"
+Demo = "https://huggingface.co/spaces/davanstrien/ocr-bench-britannica-results-qwen35-viewer"
+
+[project.scripts]
+ocr-bench = "ocr_bench.cli:main"
+
+[project.optional-dependencies]
+viewer = ["fastapi>=0.115", "uvicorn[standard]>=0.32", "jinja2>=3.1", "python-multipart>=0.0.9"]
+
+[build-system]
+requires = ["uv_build>=0.9.27,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.ruff]
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP"]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.0.2",
+    "ruff>=0.11",
+    "ty>=0.0.17",
+]

ocr_bench-0.0.1/src/ocr_bench/__init__.py

@@ -0,0 +1,3 @@
+"""OCR model evaluation toolkit — VLM-as-judge with per-dataset leaderboards."""
+
+__version__ = "0.1.0"

ocr_bench-0.0.1/src/ocr_bench/backends.py

@@ -0,0 +1,238 @@
+"""Judge backends — API-based (HF Inference Providers, OpenAI-compatible)."""
+
+from __future__ import annotations
+
+import abc
+from collections import Counter
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from typing import Any
+
+import stamina
+import structlog
+from huggingface_hub import InferenceClient
+from openai import OpenAI
+
+from ocr_bench.judge import JUDGE_SCHEMA, Comparison, parse_judge_output
+
+logger = structlog.get_logger()
+
+# Retry on these exception types with exponential backoff + jitter.
+_RETRYABLE = (Exception,)
+
+
+class JudgeBackend(abc.ABC):
+    """Base class for judge backends."""
+
+    name: str
+    concurrency: int = 1
+
+    @abc.abstractmethod
+    def _call_single(self, comp: Comparison) -> dict[str, str]:
+        """Run the judge on a single comparison."""
+
+    def judge(self, comparisons: list[Comparison]) -> list[dict[str, str]]:
+        """Run the judge on a list of comparisons (concurrently if supported).
+
+        Returns a list of parsed results (one per comparison).
+        Each result is a dict with ``winner`` and ``reason`` keys,
+        or an empty dict on failure.
+        """
+        if self.concurrency <= 1 or len(comparisons) <= 1:
+            return [self._call_single(comp) for comp in comparisons]
+
+        # Concurrent execution preserving order
+        results: list[dict[str, str]] = [{}] * len(comparisons)
+        with ThreadPoolExecutor(max_workers=self.concurrency) as pool:
+            future_to_idx = {
+                pool.submit(self._call_single, comp): i
+                for i, comp in enumerate(comparisons)
+            }
+            for future in as_completed(future_to_idx):
+                idx = future_to_idx[future]
+                try:
+                    results[idx] = future.result()
+                except Exception as exc:
+                    logger.warning("judge_call_failed", idx=idx, error=str(exc))
+                    results[idx] = {}
+        return results
+
+
+DEFAULT_MAX_TOKENS = 1024
+
+
+class InferenceProviderJudge(JudgeBackend):
+    """HF Inference Providers backend (Novita, Together, etc.)."""
+
+    def __init__(
+        self, model: str, provider: str | None = None, max_tokens: int = DEFAULT_MAX_TOKENS,
+    ):
+        self.name = f"{provider + ':' if provider else ''}{model}"
+        self.model = model
+        self.max_tokens = max_tokens
+        self.client = InferenceClient(model=model, provider=provider)  # type: ignore[invalid-argument-type]
+
+    @stamina.retry(on=_RETRYABLE, attempts=6)
+    def _call_single(self, comp: Comparison) -> dict[str, str]:
+        response = self.client.chat_completion(  # type: ignore[no-matching-overload]
+            messages=comp.messages,
+            max_tokens=self.max_tokens,
+            temperature=0.0,
+            response_format={"type": "json_object"},
+            extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+        )
+        raw = response.choices[0].message.content.strip()
+        result = parse_judge_output(raw)
+        if not result:
+            logger.warning("empty_parse", backend=self.name, sample=comp.sample_idx)
+        return result
+
+
+class OpenAICompatibleJudge(JudgeBackend):
+    """OpenAI-compatible endpoint (local vLLM server, Ollama, HF IE, etc.)."""
+
+    def __init__(
+        self,
+        base_url: str,
+        model: str = "default",
+        max_tokens: int = DEFAULT_MAX_TOKENS,
+        api_key: str = "not-needed",
+        extra_body: dict | None = None,
+        temperature: float = 0.0,
+        concurrency: int = 1,
+    ):
+        self.name = model if model != "default" else f"openai@{base_url}"
+        self.model = model
+        self.max_tokens = max_tokens
+        self.temperature = temperature
+        self.extra_body = extra_body if extra_body is not None else {"guided_json": JUDGE_SCHEMA}
+        self.concurrency = concurrency
+        self.client = OpenAI(base_url=base_url, api_key=api_key)
+
+    @stamina.retry(on=_RETRYABLE, attempts=3)
+    def _call_single(self, comp: Comparison) -> dict[str, str]:
+        response = self.client.chat.completions.create(
+            model=self.model,
+            messages=comp.messages,  # type: ignore[invalid-argument-type]
+            max_tokens=self.max_tokens,
+            temperature=self.temperature,
+            extra_body=self.extra_body,
+        )
+        raw = response.choices[0].message.content.strip()
+        result = parse_judge_output(raw)
+        if not result:
+            logger.warning("empty_parse", backend=self.name, sample=comp.sample_idx)
+        return result
+
+
+# ---------------------------------------------------------------------------
+# Spec parsing
+# ---------------------------------------------------------------------------
+
+DEFAULT_JUDGE = "novita:Qwen/Qwen3.5-35B-A3B"
+
+
+def parse_judge_spec(
+    spec: str, max_tokens: int = DEFAULT_MAX_TOKENS, concurrency: int = 1,
+) -> JudgeBackend:
+    """Parse a judge specification string into a backend.
+
+    Formats:
+      - ``"https://xxx.endpoints.huggingface.cloud"`` → :class:`OpenAICompatibleJudge`
+        (HF Inference Endpoints, OpenAI-compatible with HF token auth)
+      - ``"http://..."`` or ``"https://..."`` (other) → :class:`OpenAICompatibleJudge`
+      - ``"provider:org/model"`` (colon before first ``/``) → :class:`InferenceProviderJudge`
+      - anything else → :class:`InferenceProviderJudge` (no provider)
+    """
+    if spec.startswith("http://") or spec.startswith("https://"):
+        # Check for url:model format (e.g. https://...cloud/v1/:org/model)
+        url_part = spec
+        model_name = "default"
+        # Split on /v1/: to separate URL from model name
+        if "/v1/:" in spec:
+            url_part, model_name = spec.split("/v1/:", 1)
+            url_part += "/v1"
+
+        # HF Inference Endpoints — OpenAI-compatible, auth via HF token
+        if ".endpoints.huggingface." in url_part:
+            from huggingface_hub import get_token
+
+            base_url = url_part.rstrip("/")
+            if not base_url.endswith("/v1"):
+                base_url += "/v1"
+            token = get_token() or "not-needed"
+            return OpenAICompatibleJudge(
+                base_url=base_url,
+                model=model_name,
+                api_key=token,
+                max_tokens=max_tokens,
+                temperature=0.7,
+                extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+                concurrency=concurrency,
+            )
+        return OpenAICompatibleJudge(
+            base_url=url_part, model=model_name, max_tokens=max_tokens,
+            concurrency=concurrency,
+        )
+
+    if ":" in spec:
+        # provider:model format — colon must come before first slash
+        colon_idx = spec.index(":")
+        slash_idx = spec.find("/")
+        if slash_idx == -1 or colon_idx < slash_idx:
+            provider, model = spec.split(":", 1)
+            return InferenceProviderJudge(model=model, provider=provider, max_tokens=max_tokens)
+
+    return InferenceProviderJudge(model=spec, max_tokens=max_tokens)
+
+
+# ---------------------------------------------------------------------------
+# Jury aggregation
+# ---------------------------------------------------------------------------
+
+
+def aggregate_jury_votes(
+    all_results: list[list[dict[str, str]]],
+    judge_names: list[str],
+) -> list[dict[str, Any]]:
+    """Aggregate votes from multiple judges using majority voting.
+
+    Args:
+        all_results: List of result lists, one per judge. Each inner list
+            has one dict per comparison.
+        judge_names: Names of the judges (same order as *all_results*).
+
+    Returns:
+        Aggregated results with ``winner``, ``reason``, and ``agreement`` fields.
+    """
+    if not all_results:
+        return []
+
+    n_comparisons = len(all_results[0])
+    n_judges = len(all_results)
+    aggregated: list[dict[str, Any]] = []
+
+    for i in range(n_comparisons):
+        votes: list[str] = []
+        reasons: list[str] = []
+        for j in range(n_judges):
+            result = all_results[j][i] if i < len(all_results[j]) else {}
+            winner = result.get("winner", "")
+            if winner:
+                votes.append(winner)
+                reasons.append(f"{judge_names[j]}: {result.get('reason', '')}")
+
+        if not votes:
+            aggregated.append({"winner": "tie", "reason": "no valid votes", "agreement": "0/0"})
+            continue
+
+        counter = Counter(votes)
+        majority_winner, majority_count = counter.most_common(1)[0]
+        agreement = f"{majority_count}/{len(votes)}"
+
+        aggregated.append({
+            "winner": majority_winner,
+            "reason": "; ".join(reasons),
+            "agreement": agreement,
+        })
+
+    return aggregated