parse-bench 1.0.6__tar.gz → 2.0.0__tar.gz

{parse_bench-1.0.6 → parse_bench-2.0.0}/.gitignore

@@ -219,4 +219,5 @@ data/
 CLAUDE.md
 .DS_Store
 output/
-data_small/
+data_small/
+data_parsebench-2/

{parse_bench-1.0.6 → parse_bench-2.0.0}/CHANGELOG.md

@@ -4,14 +4,85 @@ All notable changes to `parse-bench` are recorded here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
 [Semantic Versioning](https://semver.org/).
 
-## [1.0.4] - 2026-09-04
+## [Unreleased]
+
+## [2.0.0] - 2026-11-11
+
+ParseBench 2: `--dataset parsebench-2` runs the 13 ParseBench 2 subsets and scores them into six dimensions and an
+overall score. The extended scorers are synced with the internal harness. Scores change for
+tables, formatting and layout (see Scoring); ParseBench 2 scores are not comparable with ParseBench v1
+scores.
+
+### Scoring (changes evaluation numbers)
+- `table_next`: a blank cell drawn as a dash is no longer an error wherever it sits (row label, section row, corner
+  cell), as was already the case for values; a dash written over a value is still a wrong value. A label that takes
+  another name of the reference (headers swapped or shifted, so the values sit under the wrong label) or whose number
+  changes by an inserted decimal point ("2025" read "202.5") is now a critical error, where it was read as a
+  rewording. A `th` row of column numbers that holds a formula is a header row, and a reference whose values are all
+  empty markers ("None", "N/A") is scored. About 1.5% of page scores change on saved outputs; the Tables score of a
+  system moves by about 0.1 point or less.
+- `table_next`: a whole cell holding one of the protocol's bracketed mark words reads as the mark it names:
+  `[yes]`, `[checked]`, `[icon: bullet]` as checked, `[unchecked]` / `[empty]` as unchecked, `[no]` / `[cross]` as `✗`.
+  Against a reference `●` these words scored 0.6, below an empty cell; they now score 0.95 like `✓`.
+- `formatting_v2`: a strike written with a space inside its delimiters (`~~Risk ~~Factors`) reads as struck instead of
+  plain text, and no longer pairs with a later span's `~~`. `a.`, `b)` and `i.` are list markers like `1.`, and a marker
+  inside the item's opening style (`~~a. Violation~~`) reads the same as one outside it (`a. ~~Violation~~`). Scores on
+  strikeout pages written this way rise (four audited LlamaParse pages go from 0.00–0.71 to 1.00).
+- `formatting_v2`: Markdown delimiters inside HTML table cells stay literal (`<td>**x**</td>` is not bold), and a link
+  label counts as underlined only where the reference underlines that text; elsewhere link markup is neutral.
+  `table_next`: a non-numeric cell that only partly matches is charged as "value partly differs", not "unit lost"
+  (same severity).
+- `layout_grounding_pass_rate`: a predicted box on an ignore/optional element inside a table (a checkbox mark) is
+  neutral instead of a garbage child of the table.
+- Visual Grounding attribution for PARSE results no longer depends on whether the raw provider response is
+  present. Cross-evaluation evaluated a synthetic layout result and re-resolved its adapter from it, which only
+  sees `raw_output`; LlamaParse-style adapters then lost the pages read from `ParseOutput.layout_pages` and fell
+  back to prediction-level blocks that drop chart and table content. The runner now passes the adapter that
+  produced the layout to the evaluator. Published results carry no raw response, so Visual Grounding rises for
+  providers read through the LlamaParse adapter (on LlamaParse Agentic, 57.77 -> 69.42 for the same outputs,
+  matching a local run that kept the raw response).
+- A layout page the provider returned no layout for ("<adapter> requires non-empty layout_pages") now counts as 0
+  in the aggregate, like output that cannot be adapted to a `LayoutOutput` already did. It was treated as a harness
+  error and silently dropped from the denominator, so a pipeline that returned nothing on its hardest pages scored
+  as if those pages did not exist. Visual Grounding drops for affected pipelines (median 3 of 500 pages; up to 78
+  for some lightweight models).
+- Layout adapters read `layout_pages` whose boxes are 0-1 fractions of the page, which `LayoutSegmentIR` allows
+  alongside page units. Fractional pages are scaled to page units before projection, for detection and attribution
+  alike. Without it, providers without a dedicated adapter (Surya, MinerU, Unlimited-OCR, Qianfan, LightOnOCR, GLM,
+  Mistral OCR, Nova) and Nutrient DWS localized nothing and scored 0 Visual Grounding.
+- The LlamaParse layout adapter reads `raw_output["pages"]` only when it is LlamaParse's own page payload (items with
+  `bBox`/`layoutAwareBbox`). Providers that keep a native `pages` list (Mistral OCR, GLM, Nova) now fall through to
+  their normalized `layout_pages` instead of yielding no boxes.
+- The Upstage label mapper accepts canonical labels (`Page-header`, `Text`) as well as Upstage categories, and
+  `openai_compatible_vlm_layout` (DeepSeek parse-with-layout) maps through the Qwen3-VL label indices like the other
+  parse-with-layout VLMs. Both previously raised on every page.
 
 ### Added
-- `LayoutDetectionModel.FIRECRAWL_LAYOUT` (with a `LAYOUT_MODEL_INFO` entry) so
-  Firecrawl parse output can be cross-evaluated on the layout-attribution
-  datasets, matching the other layout-capable parse providers.
+- Leaderboard accountability: `leaderboard/results.yaml` ties every `leaderboard.csv` row to its published outputs
+  in `llamaindex/ParseBench-results`, `leaderboard.csv` gains a `Verified` column, and `scripts/leaderboard/`
+  re-scores rows from those outputs. Two workflows use it: *Leaderboard verification* checks
+  leaderboard PRs against a re-score, and *Leaderboard re-score* re-scores every verified row daily when the harness
+  changes and opens a PR. See `leaderboard/README.md`.
+- `--dataset parsebench-2` on `download`, `status` and `run` for ParseBench 2 (`llamaindex/ParseBench-2`, 13 subsets,
+  downloaded to `./data_parsebench-2`). `parse-bench run <pipeline> --dataset parsebench-2` runs inference on each
+  subset folder and scores every subset with results with the extended scorers (`evaluation run --next`) into the six
+  dimensions and the overall score, written to `output/parsebench-2/_score/<pipeline>/_next_score.json`. `--group`
+  takes subset names (`charts,tables`) to run only those.
+- The `markdown_attribution` rule and its scorer (`md_attribution_v1_*` metrics, scorer version 2), ported from the
+  internal harness. ParseBench 2's `layout` subset carries one on each of its 86 `markdown_block_layout` pages,
+  which failed to load before. Its metrics are diagnostics: the Layout dimension stays `layout_grounding_pass_rate`.
+  Re-scoring the internal Azure Document Intelligence layout run with this code reproduces all of its layout and
+  attribution aggregates and every page's layout score.
 
-## [Unreleased]
+### Fixed
+- `tables_values` rules no longer crash scoring when the table is matched but too few cells match exactly.
+  That failure path returned a third tuple item, which the metric read as a score
+  (`could not convert string to float`), so the page lost every rule result instead of failing this one rule.
+  Such pages are now scored, with the rule reported as failed.
+
+## [1.0.7] - 2026-11-01
+
+Released as 1.0.5 to 1.0.7 without per-version entries; this lists what shipped by 1.0.7.
 
 ### Added
 - `repetition_overshoot`, a built-in rule that fails an output repeating anything (a leader, a unit, a token, a line,
@@ -19,21 +90,6 @@ All notable changes to `parse-bench` are recorded here. The format follows
   each of your own PDFs, budgeted from its text layer. See `docs/repetition-overshoot.md`.
 - `--dataset` on `download`, `status` and `run` accepts any HuggingFace dataset repo id (`org/name`) in the ParseBench
   layout, downloaded to `./data/<name>`.
-
-### Fixed
-- Layout attribution no longer crashes on layout rules without an `ro_index`.
-  `LayoutDetectionEvaluator` dumps rules with `ro_index=None`, which
-  `parse_gt_elements` kept, so the reading-order sort raised `TypeError` and
-  the page was dropped. A null `ro_index` now reads as `0`, as an absent key
-  already did.
-- `diagram_graph` rules no longer crash when their `reference_image` resolves.
-  The rule reached for `rules_image.thumbnail_data_uri`, a module that only
-  exists in the internal harness, so scoring a diagram with a reference render
-  raised `ModuleNotFoundError` instead of returning a score. The crash was
-  latent until 1.0.1 started forwarding the test case and source paths, which
-  is what lets the reference resolve in the first place.
-
-### Added
 - `ovisocr3_vllm` pipeline and `ovisocr3` provider for OvisOCR3
   (`OVISOCR3_SERVER_URL`). Sends the official structured-parsing prompt, re-reads
   each chart region with the official chart-parsing prompt, and scores layout
@@ -66,6 +122,33 @@ All notable changes to `parse-bench` are recorded here. The format follows
   now computed with a small numpy implementation that matches
   `sklearn.metrics.average_precision_score` (including tied-score handling),
   so evaluation numbers are unchanged. `scipy` remains a core dependency.
+- Built-in providers now register on the first `create_provider` call instead
+  of when `parse_bench.inference.providers` (or any module under it) is
+  imported. Importing `parse_bench.extensions` or the layout adapter and label
+  mapper registries no longer loads every provider and its SDK: about 0.4 s
+  instead of about 4.8 s in a full install. The provider subpackages expose
+  `load_providers()` and the registry `load_builtin_providers()`; code that
+  read `_PROVIDER_REGISTRY` directly should call the latter first.
+
+### Fixed
+- Layout attribution no longer crashes on layout rules without an `ro_index`.
+  `LayoutDetectionEvaluator` dumps rules with `ro_index=None`, which
+  `parse_gt_elements` kept, so the reading-order sort raised `TypeError` and
+  the page was dropped. A null `ro_index` now reads as `0`, as an absent key
+  already did.
+- `diagram_graph` rules no longer crash when their `reference_image` resolves.
+  The rule reached for `rules_image.thumbnail_data_uri`, a module that only
+  exists in the internal harness, so scoring a diagram with a reference render
+  raised `ModuleNotFoundError` instead of returning a score. The crash was
+  latent until 1.0.1 started forwarding the test case and source paths, which
+  is what lets the reference resolve in the first place.
+
+## [1.0.4] - 2026-09-04
+
+### Added
+- `LayoutDetectionModel.FIRECRAWL_LAYOUT` (with a `LAYOUT_MODEL_INFO` entry) so
+  Firecrawl parse output can be cross-evaluated on the layout-attribution
+  datasets, matching the other layout-capable parse providers.
 
 ## [1.0.2] - 2026-09-04
 

{parse_bench-1.0.6 → parse_bench-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: parse-bench
-Version: 1.0.6
+Version: 2.0.0
 Summary: ParseBench: a benchmark and evaluation harness for document parsing systems on real-world enterprise documents
 Project-URL: Homepage, https://parsebench.ai
 Project-URL: Repository, https://github.com/run-llama/ParseBench
@@ -38,10 +38,12 @@ Requires-Dist: markdown>=3.0
 Requires-Dist: numpy>=1.24.0
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pypdfium2>=4.30.0
 Requires-Dist: python-dateutil>=2.9.0
 Requires-Dist: python-dotenv>=1.0.0
 Requires-Dist: python-levenshtein>=0.25.0
 Requires-Dist: rapidfuzz>=3.0.0
+Requires-Dist: regex>=2024.0.0
 Requires-Dist: rich>=13.7.0
 Requires-Dist: scipy>=1.16.3
 Requires-Dist: unidecode>=1.3.0
@@ -138,42 +140,83 @@ Description-Content-Type: text/markdown
 [![Website](https://img.shields.io/badge/Website-parsebench.ai-blue)](https://parsebench.ai)
 [![arXiv](https://img.shields.io/badge/arXiv-2604.08538-b31b1b.svg)](https://arxiv.org/abs/2604.08538)
 [![Dataset](https://img.shields.io/badge/HuggingFace-Dataset-yellow)](https://huggingface.co/datasets/llamaindex/ParseBench)
+[![ParseBench 2](https://img.shields.io/badge/HuggingFace-ParseBench_2-orange)](https://huggingface.co/datasets/llamaindex/ParseBench-2)
 [![Papers with Code: SOTA on ParseBench](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge.svg?eval=32432&live=1)](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge-link?eval=32432)
 [![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
 
+> **New: [ParseBench 2](#parsebench-2)** — 4,317 test cases in six dimensions (now with forms), scored with
+> severity-aware metrics. Run it with `parse-bench run <pipeline> --dataset parsebench-2`.
+
 > **Related:** For schema-guided enterprise document extraction, see our companion benchmark [ExtractBench](https://github.com/run-llama/ExtractBench).
 
 **ParseBench** is a benchmark for evaluating how well document parsing tools convert PDFs into structured output that AI agents can reliably act on. It tests whether parsed output preserves the structure and meaning needed for autonomous decisions — not just whether it looks similar to a reference text.
 
-The benchmark covers ~2,000 human-verified pages from real enterprise documents (insurance, finance, government), organized around five capability dimensions, each targeting a failure mode that breaks production agent workflows.
+The original benchmark (ParseBench v1) covers ~2,000 human-verified pages from real enterprise documents (insurance, finance, government), organized around five capability dimensions, each targeting a failure mode that breaks production agent workflows. [ParseBench 2](#parsebench-2) extends it with more pages, a forms dimension and metrics that score each error by how much it matters.
 
 <p align="center">
   <img src="docs/parsebench_teaser.png" alt="ParseBench overview: five capability dimensions" width="100%">
 </p>
 
-## Leaderboard
+## ParseBench 2
+
+[ParseBench 2](https://huggingface.co/datasets/llamaindex/ParseBench-2) combines broad coverage of real-world documents with metrics calibrated to how much each error matters:
+
+- **Six dimensions.** Text, formatting, tables, charts, layout (visual grounding) and the new forms dimension, with 600 form pages.
+- **Severity-aware scoring.** A page score is the lower of a content-recovery score and a severity cap, so recovering most of a page cannot hide a serious error — a changed number, a flipped negation, swapped table headers or a misread form value. Changes between supported formats that keep the same information cost nothing.
+- **Calibrated against stated priorities.** The metrics are tuned against 330 controlled scenarios in six severity test suites, each with a target score drop (critical, major, minor or harmless), and check for over 140 error types.
+- **4,317 page-level test cases from 2,645 documents.** The revised ParseBench v1 pages, with more complete answer keys (every chart has its full data table), plus targeted datasets of hard cases: complex table headers, financial, insurance and dense tables, headings, strikeout, underline and government forms.
+
+One command downloads the dataset to `./data_parsebench-2`, runs a pipeline on every subset and writes the six dimension scores and the overall score to `output/parsebench-2/_score/<pipeline>/_next_score.json`:
+
+```bash
+uv run parse-bench run llamaparse_agentic --dataset parsebench-2
+
+# Only some subsets; every subset that already has results is still scored
+uv run parse-bench run llamaparse_agentic --dataset parsebench-2 --group charts,tables
+
+# Check the download and see the case count of each subset
+uv run parse-bench status --dataset parsebench-2
+```
+
+The overall score is the unweighted mean of the six dimensions. Within a dimension, page scores are averaged per dataset and the datasets are combined with fixed weights, so collecting more pages for one capability does not give it more influence:
+
+| Dimension | Datasets (`--group` name) | Cases | Metric |
+|---|---|---:|---|
+| Text | `text` | 508 | `text_next` |
+| Formatting | `text` (3/4), `headings` (1/8), `strikeout` (1/16), `underline` (1/16) | 508 + 353 | `formatting_v2` |
+| Tables | `tables` (1/2), `complex_table_headers`, `financial_tables`, `insurance_filing_tables`, `dense_tables` (1/8 each) | 1,753 | `table_next` |
+| Charts | `charts` | 568 | `chart_next` |
+| Layout | `layout` | 535 | `layout_grounding_pass_rate` |
+| Forms | `forms` (7/8), `fillable_forms` (1/8) | 600 | `forms_v2` |
+
+To score outputs from your own parser, write them in the ParseBench results layout and run `parse-bench evaluation run <dir> --next=SUBSET=RESULTS_DIR::TEST_CASES_DIR,...`. ParseBench 2 scores are not comparable with ParseBench v1 scores; the leaderboard below is ParseBench v1.
+
+## Leaderboard (ParseBench v1)
 
 <!-- LEADERBOARD:START -->
 _Top 10 by Overall score. For the full sortable, filterable leaderboard, see [parsebench.ai](https://parsebench.ai); for raw data, see [leaderboard.csv](leaderboard.csv)._
 
 | Rank | Provider | Category | Overall | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. | ¢ / Page |
 |---:|---|---|---:|---:|---:|---:|---:|---:|---:|
-| 1 | LlamaParse Agentic Plus | LlamaParse | 86.61 | 93.37 | 90.19 | 88.74 | 86.54 | 74.21 | 5.62¢ |
-| 2 | LlamaParse Agentic | LlamaParse | 83.07 | 88.88 | 81.47 | 87.63 | 84.25 | 73.12 | 1.25¢ |
-| 3 | Pulse Ultra 2 | Commercial - Startup APIs | 81.60 | 90.35 | 89.70 | 87.63 | 73.97 | 66.35 | 1.50¢ |
-| 4 | Upstage Document Parse 2 (Enhanced) | VLM - Proprietary | 80.51 | 89.49 | 77.26 | 87.36 | 76.88 | 71.56 | 3.00¢ |
-| 5 | anyformat (Standard) | Commercial - Startup APIs | 80.30 | 86.46 | 85.56 | 87.35 | 79.04 | 63.07 | 4.30¢ |
-| 6 | Anthropic Opus 5.5 (Effort High) | VLM - Proprietary | 78.89 | 94.25 | 70.89 | 91.72 | 77.04 | 60.53 | 6.13¢ |
-| 7 | oi-parser | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
-| 8 | Anthropic Fable 5.1 | VLM - Proprietary | 78.12 | 91.52 | 67.06 | 91.19 | 76.52 | 64.32 | 16.05¢ |
-| 9 | LlamaParse Cost Effective | LlamaParse | 78.02 | 85.14 | 72.99 | 87.43 | 78.29 | 66.26 | 0.38¢ |
-| 10 | rakedoc-nano | VLM - Open Weight | 77.23 | 86.44 | 64.89 | 88.84 | 71.68 | 74.28 | — |
+| 1 | LlamaParse Agentic Plus | LlamaParse | 86.69 | 93.27 | 92.19 | 89.74 | 82.84 | 75.43 | 5.62¢ |
+| 2 | LlamaParse Agentic | LlamaParse | 82.83 | 90.90 | 81.20 | 87.84 | 84.81 | 69.42 | 1.25¢ |
+| 3 | Pulse Ultra 2 † | Commercial - Startup APIs | 81.60 | 90.35 | 89.70 | 87.63 | 73.97 | 66.35 | 1.50¢ |
+| 4 | Upstage Document Parse 2 (Enhanced) | VLM - Proprietary | 80.64 | 89.48 | 77.50 | 87.68 | 76.68 | 71.84 | 3.00¢ |
+| 5 | anyformat (Standard) | Commercial - Startup APIs | 80.05 | 86.84 | 85.01 | 86.39 | 79.25 | 62.74 | 4.30¢ |
+| 6 | KDL-Frontier-Parser-nano-v2 | VLM - Open Weight | 79.07 | 86.19 | 69.14 | 87.77 | 76.91 | 75.33 | — |
+| 7 | Anthropic Opus 5.5 (Effort High) | VLM - Proprietary | 78.80 | 94.26 | 70.89 | 91.25 | 77.06 | 60.56 | 6.13¢ |
+| 8 | oi-parser † | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
+| 9 | Anthropic Fable 5.1 | VLM - Proprietary | 77.81 | 91.53 | 66.93 | 90.77 | 76.02 | 63.81 | 16.05¢ |
+| 10 | LlamaParse Cost Effective | LlamaParse | 77.68 | 86.72 | 73.98 | 87.32 | 78.14 | 62.22 | 0.38¢ |
+
+† Unverified: no published outputs in [ParseBench-results](https://huggingface.co/datasets/llamaindex/ParseBench-results); scores as submitted.
 <!-- LEADERBOARD:END -->
 
 **Inclusion criteria:**
 1. The model or API needs to be publicly accessible, either via open weights or a self-serve API that any user can sign up for.
 2. The benchmark run needs to finish within a reasonable time (roughly single-digit hours).
 3. We can adjust concurrency based on the provider's recommended settings, but providers should not require custom framework changes, so the evaluation stays fair across models.
+4. The row's per-document outputs are published to [ParseBench-results](https://huggingface.co/datasets/llamaindex/ParseBench-results). Published rows are re-scored automatically whenever the harness changes; see [leaderboard/README.md](leaderboard/README.md).
 
 ## Quick Start
 
@@ -198,6 +241,9 @@ uv run parse-bench run llamaparse_agentic --test
 # Full benchmark run (replace llamaparse_agentic with any pipeline name, see "Available Pipelines" below)
 uv run parse-bench run llamaparse_agentic
 
+# ParseBench 2: all 13 subsets, scored into six dimensions and an overall score
+uv run parse-bench run llamaparse_agentic --dataset parsebench-2
+
 # View interactive reports in your browser
 uv run parse-bench serve llamaparse_agentic
 ```
@@ -235,7 +281,7 @@ A **pipeline** is a document parsing tool or configuration that you want to eval
 
 </details>
 
-## Dataset
+## Dataset (ParseBench v1)
 
 Hosted on HuggingFace: [`llamaindex/ParseBench`](https://huggingface.co/datasets/llamaindex/ParseBench)
 
@@ -492,6 +538,7 @@ src/parse_bench/
 
 - **Paper**: [arXiv:2604.08538](https://arxiv.org/abs/2604.08538)
 - **HuggingFace Dataset**: [llamaindex/ParseBench](https://huggingface.co/datasets/llamaindex/ParseBench)
+- **ParseBench 2 Dataset**: [llamaindex/ParseBench-2](https://huggingface.co/datasets/llamaindex/ParseBench-2)
 - **Code**: [run-llama/ParseBench](https://github.com/run-llama/ParseBench)
 - **ExtractBench**: [run-llama/ExtractBench](https://github.com/run-llama/ExtractBench) - A Benchmark for Schema-Guided Enterprise Document Extraction
 

{parse_bench-1.0.6 → parse_bench-2.0.0}/README.md

@@ -3,42 +3,83 @@
 [![Website](https://img.shields.io/badge/Website-parsebench.ai-blue)](https://parsebench.ai)
 [![arXiv](https://img.shields.io/badge/arXiv-2604.08538-b31b1b.svg)](https://arxiv.org/abs/2604.08538)
 [![Dataset](https://img.shields.io/badge/HuggingFace-Dataset-yellow)](https://huggingface.co/datasets/llamaindex/ParseBench)
+[![ParseBench 2](https://img.shields.io/badge/HuggingFace-ParseBench_2-orange)](https://huggingface.co/datasets/llamaindex/ParseBench-2)
 [![Papers with Code: SOTA on ParseBench](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge.svg?eval=32432&live=1)](https://paperswithcode.co/api/v1/papers/2604.08538/leaderboard-badge-link?eval=32432)
 [![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
 
+> **New: [ParseBench 2](#parsebench-2)** — 4,317 test cases in six dimensions (now with forms), scored with
+> severity-aware metrics. Run it with `parse-bench run <pipeline> --dataset parsebench-2`.
+
 > **Related:** For schema-guided enterprise document extraction, see our companion benchmark [ExtractBench](https://github.com/run-llama/ExtractBench).
 
 **ParseBench** is a benchmark for evaluating how well document parsing tools convert PDFs into structured output that AI agents can reliably act on. It tests whether parsed output preserves the structure and meaning needed for autonomous decisions — not just whether it looks similar to a reference text.
 
-The benchmark covers ~2,000 human-verified pages from real enterprise documents (insurance, finance, government), organized around five capability dimensions, each targeting a failure mode that breaks production agent workflows.
+The original benchmark (ParseBench v1) covers ~2,000 human-verified pages from real enterprise documents (insurance, finance, government), organized around five capability dimensions, each targeting a failure mode that breaks production agent workflows. [ParseBench 2](#parsebench-2) extends it with more pages, a forms dimension and metrics that score each error by how much it matters.
 
 <p align="center">
   <img src="docs/parsebench_teaser.png" alt="ParseBench overview: five capability dimensions" width="100%">
 </p>
 
-## Leaderboard
+## ParseBench 2
+
+[ParseBench 2](https://huggingface.co/datasets/llamaindex/ParseBench-2) combines broad coverage of real-world documents with metrics calibrated to how much each error matters:
+
+- **Six dimensions.** Text, formatting, tables, charts, layout (visual grounding) and the new forms dimension, with 600 form pages.
+- **Severity-aware scoring.** A page score is the lower of a content-recovery score and a severity cap, so recovering most of a page cannot hide a serious error — a changed number, a flipped negation, swapped table headers or a misread form value. Changes between supported formats that keep the same information cost nothing.
+- **Calibrated against stated priorities.** The metrics are tuned against 330 controlled scenarios in six severity test suites, each with a target score drop (critical, major, minor or harmless), and check for over 140 error types.
+- **4,317 page-level test cases from 2,645 documents.** The revised ParseBench v1 pages, with more complete answer keys (every chart has its full data table), plus targeted datasets of hard cases: complex table headers, financial, insurance and dense tables, headings, strikeout, underline and government forms.
+
+One command downloads the dataset to `./data_parsebench-2`, runs a pipeline on every subset and writes the six dimension scores and the overall score to `output/parsebench-2/_score/<pipeline>/_next_score.json`:
+
+```bash
+uv run parse-bench run llamaparse_agentic --dataset parsebench-2
+
+# Only some subsets; every subset that already has results is still scored
+uv run parse-bench run llamaparse_agentic --dataset parsebench-2 --group charts,tables
+
+# Check the download and see the case count of each subset
+uv run parse-bench status --dataset parsebench-2
+```
+
+The overall score is the unweighted mean of the six dimensions. Within a dimension, page scores are averaged per dataset and the datasets are combined with fixed weights, so collecting more pages for one capability does not give it more influence:
+
+| Dimension | Datasets (`--group` name) | Cases | Metric |
+|---|---|---:|---|
+| Text | `text` | 508 | `text_next` |
+| Formatting | `text` (3/4), `headings` (1/8), `strikeout` (1/16), `underline` (1/16) | 508 + 353 | `formatting_v2` |
+| Tables | `tables` (1/2), `complex_table_headers`, `financial_tables`, `insurance_filing_tables`, `dense_tables` (1/8 each) | 1,753 | `table_next` |
+| Charts | `charts` | 568 | `chart_next` |
+| Layout | `layout` | 535 | `layout_grounding_pass_rate` |
+| Forms | `forms` (7/8), `fillable_forms` (1/8) | 600 | `forms_v2` |
+
+To score outputs from your own parser, write them in the ParseBench results layout and run `parse-bench evaluation run <dir> --next=SUBSET=RESULTS_DIR::TEST_CASES_DIR,...`. ParseBench 2 scores are not comparable with ParseBench v1 scores; the leaderboard below is ParseBench v1.
+
+## Leaderboard (ParseBench v1)
 
 <!-- LEADERBOARD:START -->
 _Top 10 by Overall score. For the full sortable, filterable leaderboard, see [parsebench.ai](https://parsebench.ai); for raw data, see [leaderboard.csv](leaderboard.csv)._
 
 | Rank | Provider | Category | Overall | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. | ¢ / Page |
 |---:|---|---|---:|---:|---:|---:|---:|---:|---:|
-| 1 | LlamaParse Agentic Plus | LlamaParse | 86.61 | 93.37 | 90.19 | 88.74 | 86.54 | 74.21 | 5.62¢ |
-| 2 | LlamaParse Agentic | LlamaParse | 83.07 | 88.88 | 81.47 | 87.63 | 84.25 | 73.12 | 1.25¢ |
-| 3 | Pulse Ultra 2 | Commercial - Startup APIs | 81.60 | 90.35 | 89.70 | 87.63 | 73.97 | 66.35 | 1.50¢ |
-| 4 | Upstage Document Parse 2 (Enhanced) | VLM - Proprietary | 80.51 | 89.49 | 77.26 | 87.36 | 76.88 | 71.56 | 3.00¢ |
-| 5 | anyformat (Standard) | Commercial - Startup APIs | 80.30 | 86.46 | 85.56 | 87.35 | 79.04 | 63.07 | 4.30¢ |
-| 6 | Anthropic Opus 5.5 (Effort High) | VLM - Proprietary | 78.89 | 94.25 | 70.89 | 91.72 | 77.04 | 60.53 | 6.13¢ |
-| 7 | oi-parser | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
-| 8 | Anthropic Fable 5.1 | VLM - Proprietary | 78.12 | 91.52 | 67.06 | 91.19 | 76.52 | 64.32 | 16.05¢ |
-| 9 | LlamaParse Cost Effective | LlamaParse | 78.02 | 85.14 | 72.99 | 87.43 | 78.29 | 66.26 | 0.38¢ |
-| 10 | rakedoc-nano | VLM - Open Weight | 77.23 | 86.44 | 64.89 | 88.84 | 71.68 | 74.28 | — |
+| 1 | LlamaParse Agentic Plus | LlamaParse | 86.69 | 93.27 | 92.19 | 89.74 | 82.84 | 75.43 | 5.62¢ |
+| 2 | LlamaParse Agentic | LlamaParse | 82.83 | 90.90 | 81.20 | 87.84 | 84.81 | 69.42 | 1.25¢ |
+| 3 | Pulse Ultra 2 † | Commercial - Startup APIs | 81.60 | 90.35 | 89.70 | 87.63 | 73.97 | 66.35 | 1.50¢ |
+| 4 | Upstage Document Parse 2 (Enhanced) | VLM - Proprietary | 80.64 | 89.48 | 77.50 | 87.68 | 76.68 | 71.84 | 3.00¢ |
+| 5 | anyformat (Standard) | Commercial - Startup APIs | 80.05 | 86.84 | 85.01 | 86.39 | 79.25 | 62.74 | 4.30¢ |
+| 6 | KDL-Frontier-Parser-nano-v2 | VLM - Open Weight | 79.07 | 86.19 | 69.14 | 87.77 | 76.91 | 75.33 | — |
+| 7 | Anthropic Opus 5.5 (Effort High) | VLM - Proprietary | 78.80 | 94.26 | 70.89 | 91.25 | 77.06 | 60.56 | 6.13¢ |
+| 8 | oi-parser † | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — |
+| 9 | Anthropic Fable 5.1 | VLM - Proprietary | 77.81 | 91.53 | 66.93 | 90.77 | 76.02 | 63.81 | 16.05¢ |
+| 10 | LlamaParse Cost Effective | LlamaParse | 77.68 | 86.72 | 73.98 | 87.32 | 78.14 | 62.22 | 0.38¢ |
+
+† Unverified: no published outputs in [ParseBench-results](https://huggingface.co/datasets/llamaindex/ParseBench-results); scores as submitted.
 <!-- LEADERBOARD:END -->
 
 **Inclusion criteria:**
 1. The model or API needs to be publicly accessible, either via open weights or a self-serve API that any user can sign up for.
 2. The benchmark run needs to finish within a reasonable time (roughly single-digit hours).
 3. We can adjust concurrency based on the provider's recommended settings, but providers should not require custom framework changes, so the evaluation stays fair across models.
+4. The row's per-document outputs are published to [ParseBench-results](https://huggingface.co/datasets/llamaindex/ParseBench-results). Published rows are re-scored automatically whenever the harness changes; see [leaderboard/README.md](leaderboard/README.md).
 
 ## Quick Start
 
@@ -63,6 +104,9 @@ uv run parse-bench run llamaparse_agentic --test
 # Full benchmark run (replace llamaparse_agentic with any pipeline name, see "Available Pipelines" below)
 uv run parse-bench run llamaparse_agentic
 
+# ParseBench 2: all 13 subsets, scored into six dimensions and an overall score
+uv run parse-bench run llamaparse_agentic --dataset parsebench-2
+
 # View interactive reports in your browser
 uv run parse-bench serve llamaparse_agentic
 ```
@@ -100,7 +144,7 @@ A **pipeline** is a document parsing tool or configuration that you want to eval
 
 </details>
 
-## Dataset
+## Dataset (ParseBench v1)
 
 Hosted on HuggingFace: [`llamaindex/ParseBench`](https://huggingface.co/datasets/llamaindex/ParseBench)
 
@@ -357,6 +401,7 @@ src/parse_bench/
 
 - **Paper**: [arXiv:2604.08538](https://arxiv.org/abs/2604.08538)
 - **HuggingFace Dataset**: [llamaindex/ParseBench](https://huggingface.co/datasets/llamaindex/ParseBench)
+- **ParseBench 2 Dataset**: [llamaindex/ParseBench-2](https://huggingface.co/datasets/llamaindex/ParseBench-2)
 - **Code**: [run-llama/ParseBench](https://github.com/run-llama/ParseBench)
 - **ExtractBench**: [run-llama/ExtractBench](https://github.com/run-llama/ExtractBench) - A Benchmark for Schema-Guided Enterprise Document Extraction
 

{parse_bench-1.0.6 → parse_bench-2.0.0}/pyproject.toml

@@ -36,10 +36,12 @@ dependencies = [
     "numpy>=1.24.0",
     "pandas>=2.0.0",
     "pydantic>=2.0.0",
+    "pypdfium2>=4.30.0",
     "python-dateutil>=2.9.0",
     "python-dotenv>=1.0.0",
     "python-Levenshtein>=0.25.0",
     "rapidfuzz>=3.0.0",
+    "regex>=2024.0.0",
     "rich>=13.7.0",
     "scipy>=1.16.3",
     "unidecode>=1.3.0",

{parse_bench-1.0.6 → parse_bench-2.0.0}/src/parse_bench/__init__.py

@@ -1,3 +1,3 @@
 """Document parsing evaluation system for competitive benchmarking."""
 
-__version__ = "1.0.6"
+__version__ = "2.0.0"

{parse_bench-1.0.6 → parse_bench-2.0.0}/src/parse_bench/data/cli.py

@@ -3,7 +3,7 @@
 import sys
 from pathlib import Path
 
-from parse_bench.data.download import default_data_dir, download_dataset, is_dataset_ready
+from parse_bench.data.download import dataset_spec, default_data_dir, download_dataset, is_dataset_ready
 
 
 class DataCLI:
@@ -74,6 +74,20 @@ class DataCLI:
         print(f"Dataset: {data_path}")
         print()
 
+        subsets = dataset_spec(dataset).subsets
+        if subsets:
+            hdr = f"{'Subset':<26} {'Folder':<44} {'Test Cases':>10}"
+            print(hdr)
+            print("-" * len(hdr))
+            total = 0
+            for subset, folder in subsets:
+                n_cases = sum(1 for _ in (data_path / folder).rglob("*.test.json"))
+                total += n_cases
+                print(f"{subset:<26} {folder:<44} {n_cases:>10,}")
+            print("-" * len(hdr))
+            print(f"{'Total':<26} {'':<44} {total:>10,}")
+            return 0
+
         # Gather per-category stats from JSONL files
         jsonl_files = sorted(data_path.glob("*.jsonl"))
         total_cases = 0

{parse_bench-1.0.6 → parse_bench-2.0.0}/src/parse_bench/data/download.py

@@ -1,9 +1,11 @@
 """Download parse-bench datasets from HuggingFace.
 
 Datasets (``--dataset``):
-    parsebench  https://huggingface.co/datasets/llamaindex/ParseBench (default)
-    org/name    any other HuggingFace dataset repo in the same layout: ``<category>.jsonl`` rows and ``docs/``,
-                downloaded to ``./data/<name, lowercased>``
+    parsebench    https://huggingface.co/datasets/llamaindex/ParseBench (default)
+    parsebench-2  https://huggingface.co/datasets/llamaindex/ParseBench-2, one folder per subset with a
+                  ``<name>.test.json`` next to every page, downloaded to ``./data_parsebench-2``
+    org/name      any other HuggingFace dataset repo in the same layout: ``<category>.jsonl`` rows and ``docs/``,
+                  downloaded to ``./data/<name, lowercased>``
 
 Structure after download:
     <local_dir>/
@@ -30,6 +32,28 @@ TEST_DATA_REVISION = "test-data"
 # by an existing full download.
 DEFAULT_DATA_DIR = Path("./data")
 DEFAULT_TEST_DATA_DIR = Path("./data/test")
+# Outside ./data: its sidecar test.json files would make the loader read ./data as a sidecar dataset.
+PARSEBENCH2_DATA_DIR = Path("./data_parsebench-2")
+
+# ParseBench 2 subsets (the ``evaluation run --next`` names); each one's cases are in the folder of the same name.
+PARSEBENCH2_SUBSETS: dict[str, str] = {
+    name: name
+    for name in (
+        "text",
+        "headings",
+        "strikeout",
+        "underline",
+        "tables",
+        "complex_table_headers",
+        "financial_tables",
+        "insurance_filing_tables",
+        "dense_tables",
+        "charts",
+        "layout",
+        "forms",
+        "fillable_forms",
+    )
+}
 
 
 @dataclass(frozen=True)
@@ -42,6 +66,8 @@ class DatasetSpec:
     doc_dirs: tuple[str, ...]
     test_data_dir: Path | None = None
     test_revision: str | None = None
+    # Sidecar datasets: (subset, folder) pairs; a complete copy has a test.json in every folder.
+    subsets: tuple[tuple[str, str], ...] = ()
 
 
 DATASETS: dict[str, DatasetSpec] = {
@@ -59,6 +85,13 @@ DATASETS: dict[str, DatasetSpec] = {
         test_data_dir=DEFAULT_TEST_DATA_DIR,
         test_revision=TEST_DATA_REVISION,
     ),
+    "parsebench-2": DatasetSpec(
+        repo="llamaindex/ParseBench-2",
+        data_dir=PARSEBENCH2_DATA_DIR,
+        required_files=(),
+        doc_dirs=(),
+        subsets=tuple(PARSEBENCH2_SUBSETS.items()),
+    ),
 }
 
 _REPO_ID = re.compile(r"[\w.-]+/[\w.-]+")
@@ -108,6 +141,9 @@ def is_dataset_ready(data_dir: Path, dataset: str = "parsebench") -> bool:
     if not data_dir.exists():
         return False
 
+    if spec.subsets:
+        return all(any((data_dir / folder).rglob("*.test.json")) for _, folder in spec.subsets)
+
     for f in spec.required_files:
         if not (data_dir / f).exists():
             return False

parse_bench-2.0.0/src/parse_bench/evaluation/_registration.py

@@ -0,0 +1,37 @@
+"""Shared replacement semantics for keyed layout registrations."""
+
+from collections.abc import Sequence
+from typing import Protocol
+
+
+class _KeyedRegistration(Protocol):
+    @property
+    def keys(self) -> tuple[str, ...]: ...
+
+
+def updated_registrations[RegistrationT: _KeyedRegistration](
+    registrations: Sequence[RegistrationT],
+    replacement: RegistrationT,
+    *,
+    dangerous_override: bool,
+) -> Sequence[RegistrationT]:
+    """Replace at the first matched position, retaining unrelated tie order."""
+    requested = set(replacement.keys)
+    updated: list[RegistrationT] = []
+    inserted = False
+    for entry in registrations:
+        overlap = requested.intersection(entry.keys)
+        if len(overlap) == 0:
+            updated.append(entry)
+            continue
+        if not dangerous_override:
+            raise ValueError(f"Layout implementation already registered for keys {sorted(overlap)}")
+        omitted = set(entry.keys) - requested
+        if len(omitted) > 0:
+            raise ValueError(f"Override must include all aliases of the replaced registration: {sorted(omitted)}")
+        if not inserted:
+            updated.append(replacement)
+            inserted = True
+    if not inserted:
+        updated.append(replacement)
+    return updated