fieldbench 0.2.1__tar.gz

fieldbench-0.2.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Frank Thomas
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fieldbench-0.2.1/PKG-INFO

@@ -0,0 +1,107 @@
+Metadata-Version: 2.4
+Name: fieldbench
+Version: 0.2.1
+Summary: A cross-domain, field-level benchmark for schema-driven document extraction
+Author: Frank Thomas
+License: MIT
+Project-URL: Homepage, https://github.com/fieldbench/fieldbench
+Project-URL: Corpus, https://github.com/fieldbench/corpus
+Keywords: benchmark,document-extraction,information-extraction,evaluation,nlp
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml>=6
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == "openai"
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.30; extra == "anthropic"
+Provides-Extra: llamaindex
+Requires-Dist: llama-index-core>=0.11; extra == "llamaindex"
+Requires-Dist: llama-index-llms-openai>=0.2; extra == "llamaindex"
+Provides-Extra: langchain
+Requires-Dist: langchain-openai>=0.2; extra == "langchain"
+Dynamic: license-file
+
+# FieldBench
+
+A cross-domain, field-level benchmark for **schema-driven document extraction** — and the canonical scorer that goes with it.
+
+Every document-extraction vendor claims 95%+ accuracy; almost none publish how they measure it. FieldBench makes extraction accuracy **falsifiable and comparable**: a shared corpus, a shared type-aware scorer, and a leaderboard anyone can submit to.
+
+> **Status: alpha (v0.1).** The scorer is the first piece to land. The corpus lives at [`fieldbench/corpus`](https://github.com/fieldbench/corpus). Leaderboard and HuggingFace packaging are in progress.
+
+## Install
+
+```bash
+pip install fieldbench   # once published; for now: pip install -e .
+```
+
+## Run a baseline
+
+`fieldbench run` drives any extractor over the corpus and writes the prediction files for you. It's extractor-agnostic — the whole integration surface is a `complete(prompt) -> str` callable (see `examples/openai_runner.py`):
+
+```bash
+pip install "fieldbench[openai]"
+OPENAI_API_KEY=... FIELDBENCH_MODEL=gpt-4o-mini \
+  fieldbench run --corpus ./corpus --out ./preds/gpt-4o-mini \
+    --runner examples.openai_runner:make_runner
+```
+
+Runs are **resumable** (existing predictions are skipped) and **never invent results** — a runner error writes nothing, so that document is scored as a real miss, not dropped.
+
+## Score your system
+
+Score prediction files — a flat `{field: value}` JSON named `<doc_id>.json` per document (produced by `fieldbench run` or by your own pipeline):
+
+```bash
+fieldbench score --corpus /path/to/corpus --results /path/to/predictions/
+```
+
+You get overall accuracy, the **all-null floor** (what an empty extractor scores for free), a **real-vs-synthetic** split, a **four-way outcome breakdown**, and a per-category table. `--json` emits the full report; `--category <name>` scopes to one category.
+
+## What makes the scoring type-aware
+
+`compare_field` runs **four-way null semantics** first — the thing plain accuracy hides:
+
+| Outcome | Meaning |
+|---|---|
+| `correct_absence` | field absent in GT, system correctly returned null |
+| `hallucination` | field absent in GT, system invented a value |
+| `miss` | field present in GT, system returned null |
+| `match` / `wrong_value` | both present → type-aware comparison |
+
+For present-vs-present it is tolerant where it should be and strict where it must be: numeric with cent-exact tolerance, date normalization, order-independent array F1 (partial credit), punctuation-insensitive strings, and schema enum-alias folding — but never masking a genuine content difference.
+
+## How to cite
+
+```bibtex
+@misc{fieldbench,
+  title  = {FieldBench: A Cross-Domain Benchmark for Schema-Driven Document Extraction},
+  author = {Thomas, Frank},
+  year   = {2026},
+  note   = {https://github.com/fieldbench}
+}
+```
+<!-- Replaced with the Zenodo DOI on release, and the paper citation once published. -->
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest        # golden scorer tests
+ruff check .
+```
+
+The golden tests in `tests/` are the anti-drift contract: they pin the canonical scoring semantics that any conforming implementation (including Koji's) must match.
+
+## License
+
+Code: MIT. The corpus is licensed per-source — see [`fieldbench/corpus`](https://github.com/fieldbench/corpus).

fieldbench-0.2.1/README.md

@@ -0,0 +1,75 @@
+# FieldBench
+
+A cross-domain, field-level benchmark for **schema-driven document extraction** — and the canonical scorer that goes with it.
+
+Every document-extraction vendor claims 95%+ accuracy; almost none publish how they measure it. FieldBench makes extraction accuracy **falsifiable and comparable**: a shared corpus, a shared type-aware scorer, and a leaderboard anyone can submit to.
+
+> **Status: alpha (v0.1).** The scorer is the first piece to land. The corpus lives at [`fieldbench/corpus`](https://github.com/fieldbench/corpus). Leaderboard and HuggingFace packaging are in progress.
+
+## Install
+
+```bash
+pip install fieldbench   # once published; for now: pip install -e .
+```
+
+## Run a baseline
+
+`fieldbench run` drives any extractor over the corpus and writes the prediction files for you. It's extractor-agnostic — the whole integration surface is a `complete(prompt) -> str` callable (see `examples/openai_runner.py`):
+
+```bash
+pip install "fieldbench[openai]"
+OPENAI_API_KEY=... FIELDBENCH_MODEL=gpt-4o-mini \
+  fieldbench run --corpus ./corpus --out ./preds/gpt-4o-mini \
+    --runner examples.openai_runner:make_runner
+```
+
+Runs are **resumable** (existing predictions are skipped) and **never invent results** — a runner error writes nothing, so that document is scored as a real miss, not dropped.
+
+## Score your system
+
+Score prediction files — a flat `{field: value}` JSON named `<doc_id>.json` per document (produced by `fieldbench run` or by your own pipeline):
+
+```bash
+fieldbench score --corpus /path/to/corpus --results /path/to/predictions/
+```
+
+You get overall accuracy, the **all-null floor** (what an empty extractor scores for free), a **real-vs-synthetic** split, a **four-way outcome breakdown**, and a per-category table. `--json` emits the full report; `--category <name>` scopes to one category.
+
+## What makes the scoring type-aware
+
+`compare_field` runs **four-way null semantics** first — the thing plain accuracy hides:
+
+| Outcome | Meaning |
+|---|---|
+| `correct_absence` | field absent in GT, system correctly returned null |
+| `hallucination` | field absent in GT, system invented a value |
+| `miss` | field present in GT, system returned null |
+| `match` / `wrong_value` | both present → type-aware comparison |
+
+For present-vs-present it is tolerant where it should be and strict where it must be: numeric with cent-exact tolerance, date normalization, order-independent array F1 (partial credit), punctuation-insensitive strings, and schema enum-alias folding — but never masking a genuine content difference.
+
+## How to cite
+
+```bibtex
+@misc{fieldbench,
+  title  = {FieldBench: A Cross-Domain Benchmark for Schema-Driven Document Extraction},
+  author = {Thomas, Frank},
+  year   = {2026},
+  note   = {https://github.com/fieldbench}
+}
+```
+<!-- Replaced with the Zenodo DOI on release, and the paper citation once published. -->
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest        # golden scorer tests
+ruff check .
+```
+
+The golden tests in `tests/` are the anti-drift contract: they pin the canonical scoring semantics that any conforming implementation (including Koji's) must match.
+
+## License
+
+Code: MIT. The corpus is licensed per-source — see [`fieldbench/corpus`](https://github.com/fieldbench/corpus).

fieldbench-0.2.1/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "fieldbench"
+version = "0.2.1"
+description = "A cross-domain, field-level benchmark for schema-driven document extraction"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Frank Thomas" }]
+keywords = ["benchmark", "document-extraction", "information-extraction", "evaluation", "nlp"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = ["pyyaml>=6"]
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "ruff>=0.6"]
+openai = ["openai>=1.0"]
+anthropic = ["anthropic>=0.30"]
+llamaindex = ["llama-index-core>=0.11", "llama-index-llms-openai>=0.2"]
+langchain = ["langchain-openai>=0.2"]
+
+[project.urls]
+Homepage = "https://github.com/fieldbench/fieldbench"
+Corpus = "https://github.com/fieldbench/corpus"
+
+[project.scripts]
+fieldbench = "fieldbench.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py310"

fieldbench-0.2.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

fieldbench-0.2.1/src/fieldbench/__init__.py

@@ -0,0 +1,23 @@
+"""FieldBench — a cross-domain, field-level benchmark for schema-driven document extraction."""
+
+from .aggregate import DocResult, Report, build_report
+from .corpus import list_documents, score_corpus
+from .run import LLMRunner, Runner, run_corpus
+from .scoring import SCORER_VERSION, FieldResult, compare_field
+
+__version__ = "0.2.1"
+
+__all__ = [
+    "compare_field",
+    "FieldResult",
+    "SCORER_VERSION",
+    "score_corpus",
+    "list_documents",
+    "build_report",
+    "DocResult",
+    "Report",
+    "run_corpus",
+    "Runner",
+    "LLMRunner",
+    "__version__",
+]

fieldbench-0.2.1/src/fieldbench/aggregate.py

@@ -0,0 +1,120 @@
+"""Aggregate per-field results into a FieldBench report.
+
+Reports micro/macro field accuracy, always stratified real vs synthetic, with a
+four-way confusion breakdown (correct-absence / hallucination / miss /
+wrong-value) and the all-null degenerate floor — so every score is read against
+what an empty extractor scores for free.
+"""
+
+from __future__ import annotations
+
+from collections import Counter, defaultdict
+from dataclasses import dataclass, field
+
+from .scoring import SCORER_VERSION, FieldResult, is_empty
+
+
+@dataclass
+class DocResult:
+    stem: str
+    category: str
+    source: str  # "real" | "synthetic" | "unknown"
+    fields: list[FieldResult]
+
+
+@dataclass
+class Stratum:
+    docs: int = 0
+    fields: int = 0
+    passed: int = 0
+    weighted: float = 0.0
+    four_way: Counter = field(default_factory=Counter)
+
+    @property
+    def accuracy(self) -> float:
+        return self.passed / self.fields if self.fields else 0.0
+
+    @property
+    def partial(self) -> float:
+        return self.weighted / self.fields if self.fields else 0.0
+
+    def to_dict(self) -> dict:
+        n = self.fields or 1
+        return {
+            "docs": self.docs,
+            "fields": self.fields,
+            "passed": self.passed,
+            "accuracy": round(self.accuracy, 4),
+            "partial_credit": round(self.partial, 4),
+            "four_way": {
+                bucket: {"count": c, "rate": round(c / n, 4)}
+                for bucket, c in sorted(self.four_way.items())
+            },
+        }
+
+
+@dataclass
+class Report:
+    scorer_version: str = SCORER_VERSION
+    mode: str = "unspecified"  # which representation the predictions came from
+    overall: Stratum = field(default_factory=Stratum)
+    by_source: dict[str, Stratum] = field(default_factory=dict)
+    by_category: dict[str, Stratum] = field(default_factory=dict)
+    four_way: Counter = field(default_factory=Counter)
+    all_null_floor: float = 0.0
+
+    def to_dict(self) -> dict:
+        total = sum(self.four_way.values()) or 1
+        return {
+            "scorer_version": self.scorer_version,
+            "mode": self.mode,
+            "overall": self.overall.to_dict(),
+            "by_source": {k: v.to_dict() for k, v in sorted(self.by_source.items())},
+            "by_category": {k: v.to_dict() for k, v in sorted(self.by_category.items())},
+            "four_way": {
+                bucket: {"count": n, "rate": round(n / total, 4)}
+                for bucket, n in sorted(self.four_way.items())
+            },
+            "all_null_floor": round(self.all_null_floor, 4),
+        }
+
+
+def _add(strata: dict[str, Stratum], key: str, r: FieldResult) -> None:
+    s = strata.setdefault(key, Stratum())
+    s.fields += 1
+    s.passed += int(r.passed)
+    s.weighted += r.weighted_score
+    s.four_way[r.bucket] += 1
+
+
+def build_report(docs: list[DocResult], mode: str = "unspecified") -> Report:
+    rep = Report(mode=mode)
+    empty_expected = 0
+    total = 0
+    doc_counts_source: dict[str, int] = defaultdict(int)
+    doc_counts_cat: dict[str, int] = defaultdict(int)
+
+    for doc in docs:
+        doc_counts_source[doc.source] += 1
+        doc_counts_cat[doc.category] += 1
+        for r in doc.fields:
+            total += 1
+            rep.overall.fields += 1
+            rep.overall.passed += int(r.passed)
+            rep.overall.weighted += r.weighted_score
+            rep.four_way[r.bucket] += 1
+            _add(rep.by_source, doc.source, r)
+            _add(rep.by_category, doc.category, r)
+            if is_empty(r.expected):
+                empty_expected += 1
+
+    rep.overall.docs = len(docs)
+    for src, n in doc_counts_source.items():
+        rep.by_source.setdefault(src, Stratum()).docs = n
+    for cat, n in doc_counts_cat.items():
+        rep.by_category.setdefault(cat, Stratum()).docs = n
+
+    # All-null floor: an empty extractor scores exactly the empty-GT fields
+    # (they land in `correct_absence`); everything else becomes a miss.
+    rep.all_null_floor = empty_expected / total if total else 0.0
+    return rep

fieldbench-0.2.1/src/fieldbench/cli.py

@@ -0,0 +1,134 @@
+"""FieldBench CLI: score prediction files against the corpus."""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+import json
+import sys
+from pathlib import Path
+
+from .aggregate import build_report
+from .corpus import score_corpus
+from .run import run_corpus
+
+
+def _load_runner(ref: str):
+    """Load a runner from `module:attr`. attr is a Runner instance or a
+    zero-arg factory that returns one."""
+    if ":" not in ref:
+        raise ValueError(f"--runner must be 'module:attr', got {ref!r}")
+    mod_name, attr = ref.split(":", 1)
+    obj = getattr(importlib.import_module(mod_name), attr)
+    if hasattr(obj, "extract"):
+        return obj
+    return obj()  # factory
+
+
+def _print_table(report_dict: dict) -> None:
+    o = report_dict["overall"]
+    print(f"\nFieldBench  (scorer {report_dict['scorer_version']})")
+    print("=" * 52)
+    print(f"Overall:  {o['accuracy']:.1%}  ({o['passed']}/{o['fields']} fields, {o['docs']} docs)")
+    print(f"All-null floor: {report_dict['all_null_floor']:.1%}  <- a `{{}}`-emitting baseline scores this free")
+
+    print("\nBy source:")
+    for src, s in report_dict["by_source"].items():
+        print(f"  {src:<12} {s['accuracy']:.1%}  ({s['fields']} fields, {s['docs']} docs)")
+
+    print("\nFour-way outcomes:")
+    for bucket, v in report_dict["four_way"].items():
+        print(f"  {bucket:<16} {v['count']:>6}  ({v['rate']:.1%})")
+
+    print("\nBy category:")
+    for cat, s in sorted(report_dict["by_category"].items(), key=lambda kv: kv[1]["accuracy"]):
+        print(f"  {cat:<24} {s['accuracy']:.1%}  ({s['fields']} fields)")
+    print()
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="fieldbench")
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    score = sub.add_parser("score", help="Score prediction files against the corpus")
+    score.add_argument("--corpus", required=True, type=Path, help="Path to the corpus root")
+    score.add_argument("--results", required=True, type=Path, help="Directory of <stem>.json predictions")
+    score.add_argument("--category", default=None, help="Score a single category only")
+    score.add_argument(
+        "--mode", default="unspecified", help="Label for which representation the predictions came from (markdown/source/...)"
+    )
+    score.add_argument("--fuzzy-threshold", type=float, default=0.0, help="Off (0.0) for the official metric")
+    score.add_argument("--json", action="store_true", help="Emit the full report as JSON")
+    score.add_argument(
+        "--allow-missing",
+        action="store_true",
+        help="Do not fail when prediction files are missing (they are scored as all-null)",
+    )
+
+    run = sub.add_parser("run", help="Run an extractor over the corpus, writing per-doc predictions")
+    run.add_argument("--corpus", required=True, type=Path, help="Path to the corpus root")
+    run.add_argument("--out", required=True, type=Path, help="Output dir for <stem>.json predictions")
+    run.add_argument("--runner", required=True, help="Runner as 'module:attr' (instance or zero-arg factory)")
+    run.add_argument("--mode", default="markdown", help="Which representation to feed (markdown/text/source)")
+    run.add_argument("--category", default=None, help="Run a single category only")
+    run.add_argument("--limit", type=int, default=None, help="Stop after N docs (smoke testing)")
+    run.add_argument("--no-resume", action="store_true", help="Re-run docs even if a prediction file exists")
+
+    args = parser.parse_args(argv)
+
+    if args.command == "run":
+        if not args.corpus.is_dir():
+            print(f"error: corpus not found: {args.corpus}", file=sys.stderr)
+            return 2
+        try:
+            runner = _load_runner(args.runner)
+        except (ValueError, ImportError, AttributeError) as exc:
+            print(f"error: could not load runner {args.runner!r}: {exc}", file=sys.stderr)
+            return 2
+        stats = run_corpus(
+            args.corpus,
+            runner,
+            args.out,
+            mode=args.mode,
+            category=args.category,
+            resume=not args.no_resume,
+            limit=args.limit,
+            on_progress=lambda stem, status: print(f"  {stem}: {status}", file=sys.stderr),
+        )
+        print(
+            f"\nwrote {stats.written}, skipped {stats.skipped} (resume), "
+            f"errors {stats.errors}, no-representation {stats.no_representation}"
+        )
+        if stats.error_stems:
+            print(f"error docs: {', '.join(stats.error_stems[:10])}"
+                  f"{'…' if len(stats.error_stems) > 10 else ''}", file=sys.stderr)
+        return 1 if (stats.errors or stats.no_representation) else 0
+
+    if args.command == "score":
+        if not args.corpus.is_dir():
+            print(f"error: corpus not found: {args.corpus}", file=sys.stderr)
+            return 2
+        docs, missing = score_corpus(
+            args.corpus, args.results, category=args.category, fuzzy_threshold=args.fuzzy_threshold
+        )
+        if not docs:
+            print("error: no scorable documents found", file=sys.stderr)
+            return 2
+        report = build_report(docs, mode=args.mode).to_dict()
+        if args.json:
+            print(json.dumps(report, indent=2))
+        else:
+            _print_table(report)
+        if missing and not args.allow_missing:
+            print(
+                f"error: {missing} prediction file(s) missing — scored as all-null. "
+                f"Re-run with complete results, or pass --allow-missing to override.",
+                file=sys.stderr,
+            )
+            return 1
+        return 0
+    return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())