parse-bench 1.0.4__tar.gz → 1.0.7__tar.gz

{parse_bench-1.0.4 → parse_bench-1.0.7}/CHANGELOG.md

@@ -13,11 +13,66 @@ All notable changes to `parse-bench` are recorded here. The format follows
 
 ## [Unreleased]
 
+### Added
+- `repetition_overshoot`, a built-in rule that fails an output repeating anything (a leader, a unit, a token, a line,
+  a table row, a whole table) more than the page holds. `parse-bench repetition make <dir>` writes this test next to
+  each of your own PDFs, budgeted from its text layer. See `docs/repetition-overshoot.md`.
+- `--dataset` on `download`, `status` and `run` accepts any HuggingFace dataset repo id (`org/name`) in the ParseBench
+  layout, downloaded to `./data/<name>`.
+
+### Fixed
+- Layout attribution no longer crashes on layout rules without an `ro_index`.
+  `LayoutDetectionEvaluator` dumps rules with `ro_index=None`, which
+  `parse_gt_elements` kept, so the reading-order sort raised `TypeError` and
+  the page was dropped. A null `ro_index` now reads as `0`, as an absent key
+  already did.
+- `diagram_graph` rules no longer crash when their `reference_image` resolves.
+  The rule reached for `rules_image.thumbnail_data_uri`, a module that only
+  exists in the internal harness, so scoring a diagram with a reference render
+  raised `ModuleNotFoundError` instead of returning a score. The crash was
+  latent until 1.0.1 started forwarding the test case and source paths, which
+  is what lets the reference resolve in the first place.
+
+### Added
+- `ovisocr3_vllm` pipeline and `ovisocr3` provider for OvisOCR3
+  (`OVISOCR3_SERVER_URL`). Sends the official structured-parsing prompt, re-reads
+  each chart region with the official chart-parsing prompt, and scores layout
+  through `LayoutDetectionModel.OVISOCR3_LAYOUT`.
+- `unlimitedocr2` pipeline and provider for Unlimited-OCR 2
+  (`UNLIMITEDOCR2_SERVER_URL`). Converts the model's layout-block output with
+  Baidu's official converter (vendored) and scores layout through
+  `LayoutDetectionModel.UNLIMITED_OCR_2_LAYOUT`.
+- `nemotron_parse_3_5_sdk` pipeline and `nemotron_parse_3_5` provider for
+  Nemotron 3.5 Parse, driven by an SDK server that runs the model repository's
+  triple-pass pipeline (`NEMOTRON_PARSE_3_5_SERVER_URL`). Its layout adapter
+  keeps the Canonical17 label strings under the new
+  `LayoutDetectionModel.NEMOTRON_PARSE_LAYOUT` (with a `LAYOUT_MODEL_INFO`
+  entry), and a passthrough label mapper resolves them, so Visual Grounding is
+  scored instead of raising `UnknownRawLayoutLabelError`.
+- `minicpm_ocr_sdk` pipeline and `minicpm_ocr` provider for MiniCPM-OCR 1.0,
+  driven by a server running the official `minicpmvocr` SDK
+  (`MINICPM_OCR_SERVER_URL`). PP-DocLayoutV3 regions are mapped to canonical
+  labels and scored for Visual Grounding under the new
+  `LayoutDetectionModel.MINICPM_OCR_LAYOUT` (with a `LAYOUT_MODEL_INFO` entry).
+- `register_reference_thumbnailer` (exported from `parse_bench.extensions`):
+  how a rule's `reference_image` is rendered into the detailed report. parse-bench
+  keeps no image stack in its core dependencies, so it ships no default and
+  simply omits the inline crop; a harness that already depends on one can
+  register its renderer to get crops back. A thumbnailer that raises costs the
+  report its crop, never the rule its score.
+
 ### Changed
 - Dropped the `scikit-learn` dependency. Layout-detection average precision is
   now computed with a small numpy implementation that matches
   `sklearn.metrics.average_precision_score` (including tied-score handling),
   so evaluation numbers are unchanged. `scipy` remains a core dependency.
+- Built-in providers now register on the first `create_provider` call instead
+  of when `parse_bench.inference.providers` (or any module under it) is
+  imported. Importing `parse_bench.extensions` or the layout adapter and label
+  mapper registries no longer loads every provider and its SDK: about 0.4 s
+  instead of about 4.8 s in a full install. The provider subpackages expose
+  `load_providers()` and the registry `load_builtin_providers()`; code that
+  read `_PROVIDER_REGISTRY` directly should call the latter first.
 
 ## [1.0.2] - 2026-09-04
 

{parse_bench-1.0.4 → parse_bench-1.0.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: parse-bench
-Version: 1.0.4
+Version: 1.0.7
 Summary: ParseBench: a benchmark and evaluation harness for document parsing systems on real-world enterprise documents
 Project-URL: Homepage, https://parsebench.ai
 Project-URL: Repository, https://github.com/run-llama/ParseBench
@@ -50,6 +50,8 @@ Requires-Dist: anthropic>=0.77.1; extra == 'anthropic'
 Requires-Dist: pdf2image>=1.16.0; extra == 'anthropic'
 Requires-Dist: pillow>=10.0.0; extra == 'anthropic'
 Requires-Dist: pymupdf>=1.24.0; extra == 'anthropic'
+Provides-Extra: anyformat
+Requires-Dist: httpx>=0.28.0; extra == 'anyformat'
 Provides-Extra: aws
 Requires-Dist: amazon-textract-textractor>=1.7.0; extra == 'aws'
 Requires-Dist: boto3>=1.34.0; extra == 'aws'
@@ -126,6 +128,9 @@ Requires-Dist: unstructured-client>=0.26.0; extra == 'runners'
 Requires-Dist: warp-ingest[ocr]>=2.0.1; extra == 'runners'
 Provides-Extra: unstructured
 Requires-Dist: unstructured-client>=0.26.0; extra == 'unstructured'
+Provides-Extra: upstage
+Requires-Dist: httpx>=0.28.0; extra == 'upstage'
+Requires-Dist: pillow>=10.0.0; extra == 'upstage'
 Description-Content-Type: text/markdown
 
 # ParseBench
@@ -133,6 +138,7 @@ Description-Content-Type: text/markdown
 [![Website](https://img.shields.io/badge/Website-parsebench.ai-blue)](https://parsebench.ai)
 [![arXiv](https://img.shields.io/badge/arXiv-2604.08538-b31b1b.svg)](https://arxiv.org/abs/2604.08538)
 [![Dataset](https://img.shields.io/badge/HuggingFace-Dataset-yellow)](https://huggingface.co/datasets/llamaindex/ParseBench)
+[![Papers with Code: SOTA on ParseBench](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge.svg?eval=32432&live=1)](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge-link?eval=32432)
 [![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
 
 > **Related:** For schema-guided enterprise document extraction, see our companion benchmark [ExtractBench](https://github.com/run-llama/ExtractBench).
@@ -152,16 +158,16 @@ _Top 10 by Overall score. For the full sortable, filterable leaderboard, see [pa
 
 | Rank | Provider | Category | Overall | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. | ¢ / Page |
 |---:|---|---|---:|---:|---:|---:|---:|---:|---:|
-| 1 | LlamaParse Agentic Plus | LlamaParse | 90.20 | 93.37 | 94.18 | 92.25 | 87.12 | 84.09 | 5.62¢ |
-| 2 | LlamaParse Agentic | LlamaParse | 87.01 | 88.88 | 88.68 | 91.78 | 81.44 | 84.25 | 1.25¢ |
+| 1 | LlamaParse Agentic Plus | LlamaParse | 86.61 | 93.37 | 90.19 | 88.74 | 86.54 | 74.21 | 5.62¢ |
+| 2 | LlamaParse Agentic | LlamaParse | 83.07 | 88.88 | 81.47 | 87.63 | 84.25 | 73.12 | 1.25¢ |
 | 3 | Pulse Ultra 2 | Commercial - Startup APIs | 81.60 | 90.35 | 89.70 | 87.63 | 73.97 | 66.35 | 1.50¢ |
-| 4 | LlamaParse Cost Effective | LlamaParse | 80.61 | 84.19 | 77.91 | 89.87 | 67.29 | 83.77 | 0.38¢ |
-| 5 | Anthropic Fable 5.1 | VLM - Proprietary | 78.92 | 91.52 | 67.06 | 91.19 | 76.52 | 68.3 | 16.05¢ |
-| 6 | oi-parser | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
-| 7 | rakedoc-nano | VLM - Open Weight | 77.23 | 86.44 | 64.89 | 88.84 | 71.68 | 74.28 | — |
-| 8 | florin-parser-nano | VLM - Open Weight | 76.69 | 86.10 | 65.19 | 87.37 | 70.64 | 74.14 | — |
-| 9 | KDL-Frontier-Parser-nano | VLM - Open Weight | 76.36 | 85.56 | 63.41 | 87.19 | 66.81 | 78.84 | — |
-| 10 | Extend (2.0) | Commercial - Startup APIs | 75.33 | 84.82 | 78.31 | 84.59 | 60.31 | 68.61 | 2.50¢ |
+| 4 | Upstage Document Parse 2 (Enhanced) | VLM - Proprietary | 80.51 | 89.49 | 77.26 | 87.36 | 76.88 | 71.56 | 3.00¢ |
+| 5 | anyformat (Standard) | Commercial - Startup APIs | 80.30 | 86.46 | 85.56 | 87.35 | 79.04 | 63.07 | 4.30¢ |
+| 6 | Anthropic Opus 5.5 (Effort High) | VLM - Proprietary | 78.89 | 94.25 | 70.89 | 91.72 | 77.04 | 60.53 | 6.13¢ |
+| 7 | oi-parser | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
+| 8 | Anthropic Fable 5.1 | VLM - Proprietary | 78.12 | 91.52 | 67.06 | 91.19 | 76.52 | 64.32 | 16.05¢ |
+| 9 | LlamaParse Cost Effective | LlamaParse | 78.02 | 85.14 | 72.99 | 87.43 | 78.29 | 66.26 | 0.38¢ |
+| 10 | rakedoc-nano | VLM - Open Weight | 77.23 | 86.44 | 64.89 | 88.84 | 71.68 | 74.28 | — |
 <!-- LEADERBOARD:END -->
 
 **Inclusion criteria:**
@@ -267,6 +273,23 @@ uv run parse-bench download --test
 uv run parse-bench status
 ```
 
+### Repetition tests
+
+`repetition_overshoot` is a built-in rule that fails an output repeating anything (a dot leader, a unit, a token, a
+line, a table row, a whole table) more than the page holds. Restyling, splitting, merging or dropping a repetition
+never fails; only overshoot does. An empty answer fails too.
+
+```bash
+# Write a repetition test next to each PDF, budgeted from its text layer, then run it
+uv run parse-bench repetition make ./my_docs
+uv run parse-bench run <pipeline_name> --input_dir ./my_docs
+
+# Run any HuggingFace dataset in the ParseBench layout (downloads to ./data/<name> on first run)
+uv run parse-bench run <pipeline_name> --dataset org/name
+```
+
+See [docs/repetition-overshoot.md](docs/repetition-overshoot.md) for the rule and its budgets.
+
 ## Usage
 
 ### Running the Benchmark

{parse_bench-1.0.4 → parse_bench-1.0.7}/README.md

@@ -3,6 +3,7 @@
 [![Website](https://img.shields.io/badge/Website-parsebench.ai-blue)](https://parsebench.ai)
 [![arXiv](https://img.shields.io/badge/arXiv-2604.08538-b31b1b.svg)](https://arxiv.org/abs/2604.08538)
 [![Dataset](https://img.shields.io/badge/HuggingFace-Dataset-yellow)](https://huggingface.co/datasets/llamaindex/ParseBench)
+[![Papers with Code: SOTA on ParseBench](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge.svg?eval=32432&live=1)](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge-link?eval=32432)
 [![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
 
 > **Related:** For schema-guided enterprise document extraction, see our companion benchmark [ExtractBench](https://github.com/run-llama/ExtractBench).
@@ -22,16 +23,16 @@ _Top 10 by Overall score. For the full sortable, filterable leaderboard, see [pa
 
 | Rank | Provider | Category | Overall | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. | ¢ / Page |
 |---:|---|---|---:|---:|---:|---:|---:|---:|---:|
-| 1 | LlamaParse Agentic Plus | LlamaParse | 90.20 | 93.37 | 94.18 | 92.25 | 87.12 | 84.09 | 5.62¢ |
-| 2 | LlamaParse Agentic | LlamaParse | 87.01 | 88.88 | 88.68 | 91.78 | 81.44 | 84.25 | 1.25¢ |
+| 1 | LlamaParse Agentic Plus | LlamaParse | 86.61 | 93.37 | 90.19 | 88.74 | 86.54 | 74.21 | 5.62¢ |
+| 2 | LlamaParse Agentic | LlamaParse | 83.07 | 88.88 | 81.47 | 87.63 | 84.25 | 73.12 | 1.25¢ |
 | 3 | Pulse Ultra 2 | Commercial - Startup APIs | 81.60 | 90.35 | 89.70 | 87.63 | 73.97 | 66.35 | 1.50¢ |
-| 4 | LlamaParse Cost Effective | LlamaParse | 80.61 | 84.19 | 77.91 | 89.87 | 67.29 | 83.77 | 0.38¢ |
-| 5 | Anthropic Fable 5.1 | VLM - Proprietary | 78.92 | 91.52 | 67.06 | 91.19 | 76.52 | 68.3 | 16.05¢ |
-| 6 | oi-parser | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
-| 7 | rakedoc-nano | VLM - Open Weight | 77.23 | 86.44 | 64.89 | 88.84 | 71.68 | 74.28 | — |
-| 8 | florin-parser-nano | VLM - Open Weight | 76.69 | 86.10 | 65.19 | 87.37 | 70.64 | 74.14 | — |
-| 9 | KDL-Frontier-Parser-nano | VLM - Open Weight | 76.36 | 85.56 | 63.41 | 87.19 | 66.81 | 78.84 | — |
-| 10 | Extend (2.0) | Commercial - Startup APIs | 75.33 | 84.82 | 78.31 | 84.59 | 60.31 | 68.61 | 2.50¢ |
+| 4 | Upstage Document Parse 2 (Enhanced) | VLM - Proprietary | 80.51 | 89.49 | 77.26 | 87.36 | 76.88 | 71.56 | 3.00¢ |
+| 5 | anyformat (Standard) | Commercial - Startup APIs | 80.30 | 86.46 | 85.56 | 87.35 | 79.04 | 63.07 | 4.30¢ |
+| 6 | Anthropic Opus 5.5 (Effort High) | VLM - Proprietary | 78.89 | 94.25 | 70.89 | 91.72 | 77.04 | 60.53 | 6.13¢ |
+| 7 | oi-parser | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
+| 8 | Anthropic Fable 5.1 | VLM - Proprietary | 78.12 | 91.52 | 67.06 | 91.19 | 76.52 | 64.32 | 16.05¢ |
+| 9 | LlamaParse Cost Effective | LlamaParse | 78.02 | 85.14 | 72.99 | 87.43 | 78.29 | 66.26 | 0.38¢ |
+| 10 | rakedoc-nano | VLM - Open Weight | 77.23 | 86.44 | 64.89 | 88.84 | 71.68 | 74.28 | — |
 <!-- LEADERBOARD:END -->
 
 **Inclusion criteria:**
@@ -137,6 +138,23 @@ uv run parse-bench download --test
 uv run parse-bench status
 ```
 
+### Repetition tests
+
+`repetition_overshoot` is a built-in rule that fails an output repeating anything (a dot leader, a unit, a token, a
+line, a table row, a whole table) more than the page holds. Restyling, splitting, merging or dropping a repetition
+never fails; only overshoot does. An empty answer fails too.
+
+```bash
+# Write a repetition test next to each PDF, budgeted from its text layer, then run it
+uv run parse-bench repetition make ./my_docs
+uv run parse-bench run <pipeline_name> --input_dir ./my_docs
+
+# Run any HuggingFace dataset in the ParseBench layout (downloads to ./data/<name> on first run)
+uv run parse-bench run <pipeline_name> --dataset org/name
+```
+
+See [docs/repetition-overshoot.md](docs/repetition-overshoot.md) for the rule and its budgets.
+
 ## Usage
 
 ### Running the Benchmark

{parse_bench-1.0.4 → parse_bench-1.0.7}/pyproject.toml

@@ -48,6 +48,7 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
+upstage = ["httpx>=0.28.0", "pillow>=10.0.0"]
 runners = [
     "anthropic>=0.77.1",
     "azure-ai-documentintelligence>=1.0.0",
@@ -87,6 +88,7 @@ unstructured = ["unstructured-client>=0.26.0"]
 chunkr = ["chunkr-ai>=0.0.43"]
 extend = ["extend-ai>=1.16.0"]
 docling = ["docling-core>=2.71.0", "httpx>=0.28.0"]
+anyformat = ["httpx>=0.28.0"]
 local = ["pymupdf>=1.24.0", "pypdf>=6.4.0", "pytesseract>=0.3.10", "pdf2image>=1.16.0", "pillow>=10.0.0"]
 liteparse = ["liteparse>=2.14.3"]
 pymupdf4llm = [
@@ -221,6 +223,11 @@ ignore_errors = true
 module = "parse_bench.test_cases.parse_rule_schemas"
 ignore_errors = true
 
+# Vendored verbatim from github.com/baidu/Unlimited-OCR (raw block output -> markdown).
+[[tool.mypy.overrides]]
+module = "parse_bench.inference.providers.parse._unlimitedocr2_convert"
+ignore_errors = true
+
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 python_files = ["test_*.py"]

{parse_bench-1.0.4 → parse_bench-1.0.7}/src/parse_bench/__init__.py

@@ -1,3 +1,3 @@
 """Document parsing evaluation system for competitive benchmarking."""
 
-__version__ = "1.0.4"
+__version__ = "1.0.7"

{parse_bench-1.0.4 → parse_bench-1.0.7}/src/parse_bench/cli.py

@@ -9,6 +9,7 @@ from dotenv import find_dotenv, load_dotenv
 from parse_bench import __version__
 from parse_bench.analysis.cli import AnalysisCLI
 from parse_bench.data.cli import DataCLI
+from parse_bench.data.repetition import RepetitionCLI
 from parse_bench.evaluation.cli import EvaluationCLI
 from parse_bench.inference.cli import InferenceCLI
 from parse_bench.pipeline.cli import PipelineCLI
@@ -56,6 +57,7 @@ class BenchCLI:
         analysis     Generate reports, dashboards, comparisons
         pipeline     End-to-end pipeline (same as 'run')
         data         Dataset management (same as 'download'/'status')
+        repetition   Repetition tests for your own PDFs ('parse-bench repetition make <dir>')
     """
 
     def __init__(self) -> None:
@@ -64,6 +66,7 @@ class BenchCLI:
         self.analysis = AnalysisCLI()
         self.pipeline = PipelineCLI()
         self.data = DataCLI()
+        self.repetition = RepetitionCLI()
 
     # ── Top-level convenience commands ──────────────────────────────
 
@@ -85,6 +88,7 @@ class BenchCLI:
         open_report: bool = True,
         skip_inference: bool = False,
         test: bool = False,
+        dataset: str = "parsebench",
     ) -> int:
         """Run end-to-end benchmark: inference -> evaluation -> report.
 
@@ -101,9 +105,11 @@ class BenchCLI:
             open_report: Open HTML report in browser (default: True)
             skip_inference: Skip inference, only re-evaluate (default: False)
             test: Download and run on the small test dataset (3 files per category)
+            dataset: Which dataset: 'parsebench' (default) or a HuggingFace repo id (org/name) in the same layout
 
         Example:
             parse-bench run llamaparse_agentic
+            parse-bench run llamaparse_agentic --dataset acme/MyBench
             parse-bench run llamaparse_agentic --group chart
             parse-bench run llamaparse_agentic --skip_inference
             parse-bench run llamaparse_agentic --test
@@ -121,6 +127,7 @@ class BenchCLI:
             open_report=open_report,
             skip_inference=skip_inference,
             test=test,
+            dataset=dataset,
         )
 
     def download(
@@ -128,6 +135,7 @@ class BenchCLI:
         data_dir: str | Path | None = None,
         force: bool = False,
         test: bool = False,
+        dataset: str = "parsebench",
     ) -> int:
         """Download the benchmark dataset from HuggingFace.
 
@@ -135,17 +143,20 @@ class BenchCLI:
             data_dir: Directory to store dataset (default: ./data)
             force: Force re-download even if data exists
             test: Download the small test dataset (3 files per category)
+            dataset: Which dataset: 'parsebench' (default) or a HuggingFace repo id (org/name) in the same layout
 
         Example:
             parse-bench download
             parse-bench download --test
+            parse-bench download --dataset acme/MyBench
         """
-        return self.data.download(data_dir=data_dir, force=force, test=test)
+        return self.data.download(data_dir=data_dir, force=force, test=test, dataset=dataset)
 
     def status(
         self,
         data_dir: str | Path | None = None,
         test: bool = False,
+        dataset: str = "parsebench",
     ) -> int:
         """Check if the benchmark dataset is downloaded and ready.
 
@@ -153,13 +164,15 @@ class BenchCLI:
             data_dir: Data directory to check
                 (default: ./data, or ./data/test when --test is set)
             test: Check the small test dataset instead of the full dataset
+            dataset: Which dataset: 'parsebench' (default) or a HuggingFace repo id (org/name) in the same layout
 
         Example:
             parse-bench status
             parse-bench status --test
+            parse-bench status --dataset acme/MyBench
             parse-bench status data/
         """
-        return self.data.status(data_dir=data_dir, test=test)
+        return self.data.status(data_dir=data_dir, test=test, dataset=dataset)
 
     def pipelines(self) -> None:
         """List all available pipeline configurations."""

{parse_bench-1.0.4 → parse_bench-1.0.7}/src/parse_bench/data/cli.py

@@ -14,6 +14,7 @@ class DataCLI:
         data_dir: str | Path | None = None,
         force: bool = False,
         test: bool = False,
+        dataset: str = "parsebench",
     ) -> int:
         """Download the parse-bench dataset from HuggingFace.
 
@@ -22,13 +23,14 @@ class DataCLI:
                 (default: ./data, or ./data/test when --test is set)
             force: Force re-download even if data already exists
             test: Download the small test dataset (3 files per category)
+            dataset: Which dataset: 'parsebench' (default) or a HuggingFace repo id (org/name) in the same layout
 
         Returns:
             Exit code (0 for success, non-zero for failure)
         """
         try:
-            data_path = Path(data_dir) if data_dir else default_data_dir(test=test)
-            download_dataset(data_dir=data_path, force=force, test=test)
+            data_path = Path(data_dir) if data_dir else default_data_dir(test=test, dataset=dataset)
+            download_dataset(data_dir=data_path, force=force, test=test, dataset=dataset)
             return 0
         except Exception as e:
             print(f"Error downloading dataset: {e}", file=sys.stderr)
@@ -41,6 +43,7 @@ class DataCLI:
         self,
         data_dir: str | Path | None = None,
         test: bool = False,
+        dataset: str = "parsebench",
     ) -> int:
         """Check if the dataset is downloaded and show summary statistics.
 
@@ -48,18 +51,24 @@ class DataCLI:
             data_dir: Data directory to check
                 (default: ./data, or ./data/test when --test is set)
             test: Check the small test dataset instead of the full dataset
+            dataset: Which dataset: 'parsebench' (default) or a HuggingFace repo id (org/name) in the same layout
 
         Returns:
             Exit code (0 if ready, 1 if not)
         """
         import json
 
-        data_path = Path(data_dir) if data_dir else Path.cwd() / default_data_dir(test=test)
-        ready = is_dataset_ready(data_path)
+        try:
+            data_path = Path(data_dir) if data_dir else Path.cwd() / default_data_dir(test=test, dataset=dataset)
+            ready = is_dataset_ready(data_path, dataset)
+        except ValueError as e:
+            print(f"Error: {e}", file=sys.stderr)
+            return 1
 
         if not ready:
             print(f"Dataset is NOT ready at: {data_path}")
-            print("Run 'parse-bench download' to download it.")
+            flag = "" if dataset == "parsebench" else f" --dataset {dataset}"
+            print(f"Run 'parse-bench download{flag}' to download it.")
             return 1
 
         print(f"Dataset: {data_path}")

parse_bench-1.0.7/src/parse_bench/data/download.py

@@ -0,0 +1,179 @@
+"""Download parse-bench datasets from HuggingFace.
+
+Datasets (``--dataset``):
+    parsebench  https://huggingface.co/datasets/llamaindex/ParseBench (default)
+    org/name    any other HuggingFace dataset repo in the same layout: ``<category>.jsonl`` rows and ``docs/``,
+                downloaded to ``./data/<name, lowercased>``
+
+Structure after download:
+    <local_dir>/
+    ├── chart.jsonl
+    ├── layout.jsonl
+    ├── table.jsonl
+    ├── text.jsonl
+    ├── expected_markdown.json
+    └── pdfs/{chart,layout,table,text}/*.pdf
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from pathlib import Path
+
+DATASET_REPO = "llamaindex/ParseBench"
+DATASET_REPO_TYPE = "dataset"
+TEST_DATA_REVISION = "test-data"
+
+# Default on-disk locations. Test data lives in a sibling subdirectory so the
+# two datasets coexist and `--test` does not silently overlay or get masked
+# by an existing full download.
+DEFAULT_DATA_DIR = Path("./data")
+DEFAULT_TEST_DATA_DIR = Path("./data/test")
+
+
+@dataclass(frozen=True)
+class DatasetSpec:
+    """A dataset the CLI can download: where it lives on HuggingFace and on disk, and what a complete copy holds."""
+
+    repo: str
+    data_dir: Path
+    required_files: tuple[str, ...]
+    doc_dirs: tuple[str, ...]
+    test_data_dir: Path | None = None
+    test_revision: str | None = None
+
+
+DATASETS: dict[str, DatasetSpec] = {
+    "parsebench": DatasetSpec(
+        repo=DATASET_REPO,
+        data_dir=DEFAULT_DATA_DIR,
+        required_files=(
+            "chart.jsonl",
+            "layout.jsonl",
+            "table.jsonl",
+            "text_content.jsonl",
+            "text_formatting.jsonl",
+        ),
+        doc_dirs=("docs/chart", "docs/layout", "docs/table", "docs/text"),
+        test_data_dir=DEFAULT_TEST_DATA_DIR,
+        test_revision=TEST_DATA_REVISION,
+    ),
+}
+
+_REPO_ID = re.compile(r"[\w.-]+/[\w.-]+")
+
+
+def dataset_spec(dataset: str = "parsebench", test: bool = False) -> DatasetSpec:
+    """Look up a dataset by name or HuggingFace repo id; ``test`` requires one that has a test subset.
+
+    A repo id (``org/name``) not in ``DATASETS`` is any dataset in the ParseBench layout. Its categories are not
+    known in advance, so a complete copy is one with at least one ``*.jsonl`` and a non-empty ``docs/``.
+    """
+    if dataset in DATASETS:
+        spec = DATASETS[dataset]
+    elif _REPO_ID.fullmatch(dataset):
+        name = dataset.split("/", 1)[1].lower()
+        spec = DatasetSpec(repo=dataset, data_dir=DEFAULT_DATA_DIR / name, required_files=(), doc_dirs=("docs",))
+    else:
+        raise ValueError(
+            f"Unknown dataset {dataset!r}; choose one of: {', '.join(DATASETS)}, or a HuggingFace repo id (org/name)"
+        )
+    if test and spec.test_data_dir is None:
+        raise ValueError(f"The {dataset!r} dataset has no test subset; drop --test")
+    return spec
+
+
+def default_data_dir(test: bool = False, dataset: str = "parsebench") -> Path:
+    """Return the default on-disk dataset path for the given mode.
+
+    Used by every CLI surface that takes ``--test`` so the routing is
+    consistent between ``download``, ``run`` and ``status``.
+    """
+    spec = dataset_spec(dataset, test)
+    return spec.test_data_dir if test and spec.test_data_dir else spec.data_dir
+
+
+def is_dataset_ready(data_dir: Path, dataset: str = "parsebench") -> bool:
+    """Check if the dataset is already downloaded and complete.
+
+    Args:
+        data_dir: Path to the data directory.
+        dataset: Which dataset to check for (see ``dataset_spec``).
+
+    Returns:
+        True if all required files and at least one PDF per category exist.
+    """
+    spec = dataset_spec(dataset)
+    if not data_dir.exists():
+        return False
+
+    for f in spec.required_files:
+        if not (data_dir / f).exists():
+            return False
+    if not spec.required_files and not any(data_dir.glob("*.jsonl")):
+        return False
+
+    for d in spec.doc_dirs:
+        doc_dir = data_dir / d
+        if not doc_dir.exists():
+            return False
+        # Check for any supported document file (PDF, image, etc.)
+        if not any(doc_dir.rglob("*.*")):
+            return False
+
+    return True
+
+
+def download_dataset(
+    data_dir: Path | None = None,
+    force: bool = False,
+    test: bool = False,
+    dataset: str = "parsebench",
+) -> Path:
+    """Download a parse-bench dataset from HuggingFace.
+
+    Uses huggingface_hub's snapshot_download to fetch the full dataset,
+    including JSONL files and PDFs.
+
+    Args:
+        data_dir: Local directory to store the dataset.
+            Defaults to ./data in the current working directory.
+        force: Force re-download even if data already exists.
+        test: Download the small test dataset (3 files per category)
+            instead of the full dataset.
+        dataset: Which dataset to download: a name in ``DATASETS`` or a HuggingFace repo id.
+
+    Returns:
+        Path to the downloaded dataset directory.
+    """
+    from huggingface_hub import snapshot_download
+
+    spec = dataset_spec(dataset, test)
+    if data_dir is None:
+        data_dir = Path.cwd() / default_data_dir(test=test, dataset=dataset)
+
+    revision = spec.test_revision if test else None
+
+    if not force and is_dataset_ready(data_dir, dataset):
+        print(f"Dataset already downloaded at: {data_dir}")
+        return data_dir
+
+    label = "test dataset" if test else "dataset"
+    print(f"Downloading {label} from HuggingFace: {spec.repo}")
+    if revision:
+        print(f"Branch: {revision}")
+    print(f"Destination: {data_dir}")
+
+    snapshot_download(
+        repo_id=spec.repo,
+        repo_type=DATASET_REPO_TYPE,
+        local_dir=str(data_dir),
+        revision=revision,
+    )
+
+    if not is_dataset_ready(data_dir, dataset):
+        raise RuntimeError(f"Dataset download completed but validation failed. Check {data_dir} for missing files.")
+
+    print(f"Dataset ready at: {data_dir}")
+    return data_dir