geclens 1.0.0__tar.gz

geclens-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Prakhar Gautam, Jitendra Singh Thakur, Ashish Mishra
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

geclens-1.0.0/PKG-INFO

@@ -0,0 +1,202 @@
+Metadata-Version: 2.4
+Name: geclens
+Version: 1.0.0
+Summary: Error-type-targeted evaluation of grammatical error correction systems
+Author: Jitendra Singh Thakur, Ashish Mishra
+Author-email: Prakhar Gautam <prakhargautam@jecjabalpur.ac.in>
+License: MIT
+Project-URL: Homepage, https://github.com/prakhargautam-ggits/GECLens
+Project-URL: Documentation, https://prakhargautam-ggits.github.io/GECLens/
+Project-URL: Issues, https://github.com/prakhargautam-ggits/GECLens/issues
+Keywords: grammatical error correction,evaluation,ERRANT,error types,significance testing,NLP
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Intended Audience :: Science/Research
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: errant==3.0.2
+Requires-Dist: spacy<3.8,>=3.4
+Requires-Dist: numpy>=1.21
+Requires-Dist: matplotlib>=3.5
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: click>=8.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov>=4; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.5; extra == "docs"
+Requires-Dist: mkdocs-material>=9; extra == "docs"
+Dynamic: license-file
+
+# GECLens
+
+**Error-type-targeted evaluation of grammatical error correction (GEC) systems.**
+
+[![tests](https://github.com/prakhargautam-ggits/GECLens/actions/workflows/ci.yml/badge.svg)](https://github.com/prakhargautam-ggits/GECLens/actions/workflows/ci.yml)
+![python](https://img.shields.io/badge/python-3.9%E2%80%933.12-blue) ![license](https://img.shields.io/badge/license-MIT-green)
+
+Overall F0.5 hides how a system handles a single kind of error. Subject–verb agreement, articles, prepositions and tense
+each make up only a few per cent of all edits. GECLens answers two questions for any ERRANT error type:
+
+1. **Does the system fix these errors?** Type F0.5, type-only F0.5, sentence-level detection and correction success.
+2. **What does it break while trying?** Over-correction on sentences the references leave unchanged.
+
+Every measure comes with bootstrap confidence intervals and paired significance tests (approximate randomisation with
+Holm correction), for any number of systems. Results are written as CSV, Markdown and LaTeX tables and as figures.
+
+GECLens builds on [ERRANT](https://github.com/chrisjbryant/errant) for alignment and edit typing. Its edit-level scores
+are **identical** to `errant_compare`. This is checked in the test suite.
+
+---
+
+## Installation
+
+```bash
+pip install geclens            # or, from source: pip install -e .
+python -m spacy download en_core_web_sm
+```
+
+Requirements: Python 3.9–3.12, ERRANT 3.0.2, spaCy 3.4–3.7 with `en_core_web_sm`, NumPy, Matplotlib and PyYAML. GECLens
+runs on Linux, macOS and Windows.
+
+## Quick start (toy data included)
+
+```bash
+cd examples/toy
+geclens run config.yaml          # about 5 seconds
+```
+
+This scores three toy systems on 30 sentences for three error types and writes `results/`. The toy data were written by
+the GECLens authors and are released under CC0.
+
+| System | Overall F0.5 | Type F0.5 | Type-only F0.5 | Detection F1 | Success | Over-corr. |
+|---|---|---|---|---|---|---|
+| careful | 0.882 | 0.972 | 0.972 | 0.933 | 87.5% | 0.0% |
+| eager | 0.649 | 0.987 | 0.987 | 0.968 | 93.8% | 100.0% |
+| cautious | 0.312 | 0.435 | 0.435 | 0.222 | 12.5% | 0.0% |
+
+*eager* fixes agreement errors as well as *careful*, but it also rewrites every correct sentence. Type-level F0.5 alone
+would not show this.
+
+## What GECLens measures
+
+For an error-type selector **T** (for example `R:VERB:SVA`):
+
+| Measure | Definition |
+|---|---|
+| `overall` | ERRANT precision, recall and F0.5 over all edits |
+| `type` | ERRANT P/R/F0.5 for edits of type T, for the annotator ERRANT selects per sentence (the one that maximises overall F0.5) |
+| `type_only` | P/R/F0.5 after restricting hypothesis **and** reference edits to T, so the annotator choice depends on T edits only. Use it to compare systems that make different amounts of other edits. |
+| `detection` | Sentence-level P/R/F1 for "the sentence contains a T error" |
+| `success` | Share of T-positive sentences where every T edit of at least one annotator was produced exactly |
+| `overcorrection` | Share of **unedited** reference sentences that the system changed (whitespace and quote style ignored) |
+| `change_rate` | Share of all sentences the system changed |
+
+**Subsets** (`geclens subset`): `positive` (some annotator made a T edit), `positive_all`, `unedited` (no annotator made
+any edit), `unedited_any`, and `other`.
+
+**Error-type selectors:**
+- an exact type (`R:VERB:SVA`);
+- a category, which matches its M/R/U variants (`DET`);
+- a pattern (`R:VERB*`);
+- a comma-separated list (`PREP,DET`).
+
+## Command line
+
+```bash
+# subsets and reference statistics
+geclens subset  --ref ref.m2 --type R:VERB:SVA --out subsets.json
+
+# scores with 95% bootstrap confidence intervals
+geclens score   --src src.txt --ref ref.m2 --type DET \
+                --hyp sysA=a.txt sysB=b.txt --ci 2000 --out-dir results
+
+# paired approximate randomisation, Holm-adjusted, every system vs a baseline
+geclens compare --src src.txt --ref ref.m2 --type PREP --hyp sysA=a.txt sysB=b.txt base=c.txt \
+                --baseline base --measures type_only overcorrection --shuffles 10000
+
+# system x error-type heatmap of ERRANT F0.5
+geclens profile --src src.txt --ref ref.m2 --hyp a.txt b.txt c.txt --group category --top 12
+
+# everything from one configuration file
+geclens run config.yaml
+```
+
+Inputs:
+- `--src`: tokenised source sentences, one per line.
+- `--ref`: an ERRANT-typed reference M2 file. If you only have corrected text, create it with
+  `errant_parallel -orig src.txt -cor ref1.txt [ref2.txt ...] -out ref.m2`.
+- `--hyp`: system outputs as text (one sentence per line) or as ready-made `.m2` files.
+
+Hypothesis M2 files are built once with ERRANT and cached.
+
+## Python API
+
+```python
+from geclens import score_many, bootstrap_ci, compare_systems, profile_table, evaluate
+
+scores = score_many("src.txt", "ref.m2", {"A": "a.txt", "B": "b.txt"}, selector="R:VERB:SVA")
+scores["A"].metrics["type_only"]                    # {'tp':…, 'fp':…, 'fn':…, 'P':…, 'R':…, 'F0.5':…}
+bootstrap_ci(scores["A"], resamples=2000)           # {'overall': (lo, hi), 'type': …, …}
+compare_systems(scores, ["type_only", "overcorrection"], baseline="B")
+
+evaluate("src.txt", "ref.m2", {"A": "a.txt", "B": "b.txt"}, selector="PREP", out_dir="results")
+```
+
+## Configuration file
+
+```yaml
+source: src.txt
+reference: ref.m2
+out_dir: results
+selectors: [R:VERB:SVA, PREP, DET]      # one evaluation per selector
+systems: {careful: careful.txt, eager: eager.txt, cautious: cautious.txt}
+statistics: {bootstrap_resamples: 2000, shuffles: 10000, measures: [type_only, overcorrection], baseline: cautious, seed: 13}
+profile: {group: category, top: 12}
+```
+
+## Output files
+
+| File | Content |
+|---|---|
+| `scores.csv` / `.md` / `.tex` | One row per system; the Markdown and LaTeX tables include 95% CIs |
+| `<system>.json` | All measures and per-type ERRANT counts |
+| `confidence_intervals.json` | Bootstrap intervals |
+| `comparisons.md` / `.json` | Differences, paired-bootstrap CIs, p-values and Holm-adjusted p-values |
+| `scatter.png` | Type-only F0.5 against over-correction |
+| `profile.png` / `.json` | System × error-type heatmap |
+| `subsets.json`, `reference_stats.json` | Subset indices and reference statistics |
+
+## Correctness
+
+- Hypothesis M2 files are byte-identical to `errant_parallel`, overall and per-type TP/FP/FN equal
+  `errant_compare -cat 3`, and the per-sentence counts equal those of the annotator that `errant_compare -v` reports
+  as chosen for each sentence (see `tests/test_score.py`
+  and `tests/test_edge_cases.py`). Check this on your own data with
+  `python scripts/check_against_errant.py src.txt ref.m2 sys1.txt [sys2.txt ...]`.
+- A hand-built case with two annotators covers missed, unnecessary and replacement edits, two target edits in one
+  sentence, an unedited sentence and an empty subset; expected values were worked out by hand.
+- Statistical routines are tested on known cases: identical systems give p = 1, Holm values are checked by hand, and
+  bootstrap intervals contain the point estimate.
+- The CI runs the suite on Python 3.9–3.12.
+
+```bash
+pip install -e ".[dev]" && pytest
+```
+
+## Citation
+
+If you use GECLens, please cite the software (see `CITATION.cff`):
+
+> P. Gautam, J. S. Thakur, A. Mishra. *GECLens: A Python toolkit for error-type-targeted evaluation of grammatical error
+> correction systems.* Version 1.0.0, 2026. https://github.com/prakhargautam-ggits/GECLens
+
+## Licence and support
+
+- MIT licence (`LICENSE`, `Licence.txt`).
+- Questions and bug reports: [GitHub issues](https://github.com/prakhargautam-ggits/GECLens/issues) or
+  prakhargautam@jecjabalpur.ac.in.

geclens-1.0.0/README.md

@@ -0,0 +1,168 @@
+# GECLens
+
+**Error-type-targeted evaluation of grammatical error correction (GEC) systems.**
+
+[![tests](https://github.com/prakhargautam-ggits/GECLens/actions/workflows/ci.yml/badge.svg)](https://github.com/prakhargautam-ggits/GECLens/actions/workflows/ci.yml)
+![python](https://img.shields.io/badge/python-3.9%E2%80%933.12-blue) ![license](https://img.shields.io/badge/license-MIT-green)
+
+Overall F0.5 hides how a system handles a single kind of error. Subject–verb agreement, articles, prepositions and tense
+each make up only a few per cent of all edits. GECLens answers two questions for any ERRANT error type:
+
+1. **Does the system fix these errors?** Type F0.5, type-only F0.5, sentence-level detection and correction success.
+2. **What does it break while trying?** Over-correction on sentences the references leave unchanged.
+
+Every measure comes with bootstrap confidence intervals and paired significance tests (approximate randomisation with
+Holm correction), for any number of systems. Results are written as CSV, Markdown and LaTeX tables and as figures.
+
+GECLens builds on [ERRANT](https://github.com/chrisjbryant/errant) for alignment and edit typing. Its edit-level scores
+are **identical** to `errant_compare`. This is checked in the test suite.
+
+---
+
+## Installation
+
+```bash
+pip install geclens            # or, from source: pip install -e .
+python -m spacy download en_core_web_sm
+```
+
+Requirements: Python 3.9–3.12, ERRANT 3.0.2, spaCy 3.4–3.7 with `en_core_web_sm`, NumPy, Matplotlib and PyYAML. GECLens
+runs on Linux, macOS and Windows.
+
+## Quick start (toy data included)
+
+```bash
+cd examples/toy
+geclens run config.yaml          # about 5 seconds
+```
+
+This scores three toy systems on 30 sentences for three error types and writes `results/`. The toy data were written by
+the GECLens authors and are released under CC0.
+
+| System | Overall F0.5 | Type F0.5 | Type-only F0.5 | Detection F1 | Success | Over-corr. |
+|---|---|---|---|---|---|---|
+| careful | 0.882 | 0.972 | 0.972 | 0.933 | 87.5% | 0.0% |
+| eager | 0.649 | 0.987 | 0.987 | 0.968 | 93.8% | 100.0% |
+| cautious | 0.312 | 0.435 | 0.435 | 0.222 | 12.5% | 0.0% |
+
+*eager* fixes agreement errors as well as *careful*, but it also rewrites every correct sentence. Type-level F0.5 alone
+would not show this.
+
+## What GECLens measures
+
+For an error-type selector **T** (for example `R:VERB:SVA`):
+
+| Measure | Definition |
+|---|---|
+| `overall` | ERRANT precision, recall and F0.5 over all edits |
+| `type` | ERRANT P/R/F0.5 for edits of type T, for the annotator ERRANT selects per sentence (the one that maximises overall F0.5) |
+| `type_only` | P/R/F0.5 after restricting hypothesis **and** reference edits to T, so the annotator choice depends on T edits only. Use it to compare systems that make different amounts of other edits. |
+| `detection` | Sentence-level P/R/F1 for "the sentence contains a T error" |
+| `success` | Share of T-positive sentences where every T edit of at least one annotator was produced exactly |
+| `overcorrection` | Share of **unedited** reference sentences that the system changed (whitespace and quote style ignored) |
+| `change_rate` | Share of all sentences the system changed |
+
+**Subsets** (`geclens subset`): `positive` (some annotator made a T edit), `positive_all`, `unedited` (no annotator made
+any edit), `unedited_any`, and `other`.
+
+**Error-type selectors:**
+- an exact type (`R:VERB:SVA`);
+- a category, which matches its M/R/U variants (`DET`);
+- a pattern (`R:VERB*`);
+- a comma-separated list (`PREP,DET`).
+
+## Command line
+
+```bash
+# subsets and reference statistics
+geclens subset  --ref ref.m2 --type R:VERB:SVA --out subsets.json
+
+# scores with 95% bootstrap confidence intervals
+geclens score   --src src.txt --ref ref.m2 --type DET \
+                --hyp sysA=a.txt sysB=b.txt --ci 2000 --out-dir results
+
+# paired approximate randomisation, Holm-adjusted, every system vs a baseline
+geclens compare --src src.txt --ref ref.m2 --type PREP --hyp sysA=a.txt sysB=b.txt base=c.txt \
+                --baseline base --measures type_only overcorrection --shuffles 10000
+
+# system x error-type heatmap of ERRANT F0.5
+geclens profile --src src.txt --ref ref.m2 --hyp a.txt b.txt c.txt --group category --top 12
+
+# everything from one configuration file
+geclens run config.yaml
+```
+
+Inputs:
+- `--src`: tokenised source sentences, one per line.
+- `--ref`: an ERRANT-typed reference M2 file. If you only have corrected text, create it with
+  `errant_parallel -orig src.txt -cor ref1.txt [ref2.txt ...] -out ref.m2`.
+- `--hyp`: system outputs as text (one sentence per line) or as ready-made `.m2` files.
+
+Hypothesis M2 files are built once with ERRANT and cached.
+
+## Python API
+
+```python
+from geclens import score_many, bootstrap_ci, compare_systems, profile_table, evaluate
+
+scores = score_many("src.txt", "ref.m2", {"A": "a.txt", "B": "b.txt"}, selector="R:VERB:SVA")
+scores["A"].metrics["type_only"]                    # {'tp':…, 'fp':…, 'fn':…, 'P':…, 'R':…, 'F0.5':…}
+bootstrap_ci(scores["A"], resamples=2000)           # {'overall': (lo, hi), 'type': …, …}
+compare_systems(scores, ["type_only", "overcorrection"], baseline="B")
+
+evaluate("src.txt", "ref.m2", {"A": "a.txt", "B": "b.txt"}, selector="PREP", out_dir="results")
+```
+
+## Configuration file
+
+```yaml
+source: src.txt
+reference: ref.m2
+out_dir: results
+selectors: [R:VERB:SVA, PREP, DET]      # one evaluation per selector
+systems: {careful: careful.txt, eager: eager.txt, cautious: cautious.txt}
+statistics: {bootstrap_resamples: 2000, shuffles: 10000, measures: [type_only, overcorrection], baseline: cautious, seed: 13}
+profile: {group: category, top: 12}
+```
+
+## Output files
+
+| File | Content |
+|---|---|
+| `scores.csv` / `.md` / `.tex` | One row per system; the Markdown and LaTeX tables include 95% CIs |
+| `<system>.json` | All measures and per-type ERRANT counts |
+| `confidence_intervals.json` | Bootstrap intervals |
+| `comparisons.md` / `.json` | Differences, paired-bootstrap CIs, p-values and Holm-adjusted p-values |
+| `scatter.png` | Type-only F0.5 against over-correction |
+| `profile.png` / `.json` | System × error-type heatmap |
+| `subsets.json`, `reference_stats.json` | Subset indices and reference statistics |
+
+## Correctness
+
+- Hypothesis M2 files are byte-identical to `errant_parallel`, overall and per-type TP/FP/FN equal
+  `errant_compare -cat 3`, and the per-sentence counts equal those of the annotator that `errant_compare -v` reports
+  as chosen for each sentence (see `tests/test_score.py`
+  and `tests/test_edge_cases.py`). Check this on your own data with
+  `python scripts/check_against_errant.py src.txt ref.m2 sys1.txt [sys2.txt ...]`.
+- A hand-built case with two annotators covers missed, unnecessary and replacement edits, two target edits in one
+  sentence, an unedited sentence and an empty subset; expected values were worked out by hand.
+- Statistical routines are tested on known cases: identical systems give p = 1, Holm values are checked by hand, and
+  bootstrap intervals contain the point estimate.
+- The CI runs the suite on Python 3.9–3.12.
+
+```bash
+pip install -e ".[dev]" && pytest
+```
+
+## Citation
+
+If you use GECLens, please cite the software (see `CITATION.cff`):
+
+> P. Gautam, J. S. Thakur, A. Mishra. *GECLens: A Python toolkit for error-type-targeted evaluation of grammatical error
+> correction systems.* Version 1.0.0, 2026. https://github.com/prakhargautam-ggits/GECLens
+
+## Licence and support
+
+- MIT licence (`LICENSE`, `Licence.txt`).
+- Questions and bug reports: [GitHub issues](https://github.com/prakhargautam-ggits/GECLens/issues) or
+  prakhargautam@jecjabalpur.ac.in.

geclens-1.0.0/geclens/__init__.py

@@ -0,0 +1,14 @@
+"""GECLens: error-type-targeted evaluation of grammatical error correction systems."""
+
+__version__ = "1.0.0"
+
+from .types import TypeSelector  # noqa: E402,F401
+from .data import Subsets, build_subsets, reference_stats  # noqa: E402,F401
+from .score import SystemScore, score_system  # noqa: E402,F401
+from .stats import approx_randomisation, bootstrap_ci, compare_systems, holm, paired_bootstrap_ci  # noqa: E402,F401
+from .profile import profile_table  # noqa: E402,F401
+from .pipeline import evaluate, score_many  # noqa: E402,F401
+
+__all__ = ["TypeSelector", "Subsets", "build_subsets", "reference_stats", "SystemScore", "score_system",
+           "approx_randomisation", "bootstrap_ci", "compare_systems", "holm", "paired_bootstrap_ci", "profile_table",
+           "evaluate", "score_many", "__version__"]

geclens-1.0.0/geclens/align.py

@@ -0,0 +1,75 @@
+"""Convert system output text into hypothesis M2 with ERRANT.
+
+The loop follows ``errant_parallel`` line by line, so the resulting M2 is identical to the command-line tool. The ERRANT
+annotator (spaCy model) is loaded once and reused for every system, and results are cached on disk by content hash.
+"""
+from __future__ import annotations
+
+import hashlib
+import os
+from typing import List, Optional
+
+from .m2 import NOOP_LINE, read_lines
+
+_ANNOTATOR = None
+
+
+def get_annotator(lang: str = "en"):
+    global _ANNOTATOR
+    if _ANNOTATOR is None:
+        import errant  # imported lazily: loading spaCy takes a few seconds
+
+        _ANNOTATOR = errant.load(lang)
+    return _ANNOTATOR
+
+
+def align_lines(src: List[str], hyp: List[str], tokenise: bool = False, lev: bool = False,
+                merge: str = "rules", annotator=None) -> str:
+    """Return the M2 text for one system (one annotator, id 0)."""
+    if len(src) != len(hyp):
+        raise ValueError(f"source has {len(src)} lines but the system output has {len(hyp)}")
+    ann = annotator or get_annotator()
+    out = []
+    for i, (o, c) in enumerate(zip(src, hyp)):
+        o, c = o.strip(), c.strip()
+        if not o:
+            raise ValueError(f"empty source line {i + 1}: ERRANT would skip it and misalign the files")
+        orig = ann.parse(o, tokenise)
+        lines = ["S " + " ".join(t.text for t in orig)]
+        if orig.text.strip() == c:
+            lines.append(NOOP_LINE.format(a=0))
+        else:
+            cor = ann.parse(c, tokenise)
+            for edit in ann.annotate(orig, cor, lev, merge):
+                lines.append(edit.to_m2(0))
+        out.append("\n".join(lines))
+    return "".join(block + "\n\n" for block in out)  # byte-identical to errant_parallel
+
+
+def _digest(*parts: str) -> str:
+    h = hashlib.sha256()
+    for p in parts:
+        h.update(p.encode("utf-8"))
+        h.update(b"\0")
+    return h.hexdigest()[:20]
+
+
+def hypothesis_m2(src_path: str, hyp_path: str, cache_dir: Optional[str] = None, tokenise: bool = False) -> str:
+    """Path of the hypothesis M2 for ``hyp_path``; built with ERRANT (and cached) if needed.
+
+    If ``hyp_path`` already ends in ``.m2`` it is returned unchanged.
+    """
+    if hyp_path.endswith(".m2"):
+        return hyp_path
+    src, hyp = read_lines(src_path), read_lines(hyp_path)
+    cache_dir = cache_dir or os.path.join(os.path.dirname(os.path.abspath(hyp_path)), ".geclens_cache")
+    os.makedirs(cache_dir, exist_ok=True)
+    key = _digest("align-v2", "\n".join(src), "\n".join(hyp), str(tokenise))
+    out = os.path.join(cache_dir, f"{os.path.splitext(os.path.basename(hyp_path))[0]}.{key}.m2")
+    if not os.path.exists(out):
+        text = align_lines(src, hyp, tokenise=tokenise)
+        tmp = out + ".tmp"
+        with open(tmp, "w", encoding="utf-8") as fh:
+            fh.write(text)
+        os.replace(tmp, out)
+    return out

geclens-1.0.0/geclens/cli.py

@@ -0,0 +1,131 @@
+"""Command-line interface: ``geclens subset | score | compare | profile | run``."""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from typing import Dict, List
+
+from . import __version__, report
+from .data import build_subsets, reference_stats
+
+
+def _systems(items: List[str]) -> Dict[str, str]:
+    out = {}
+    for it in items:
+        if "=" in it:
+            name, path = it.split("=", 1)
+        else:
+            path = it
+            name = os.path.splitext(os.path.basename(path))[0]
+        if name in out:
+            raise SystemExit(f"duplicate system name: {name}")
+        out[name] = path
+    return out
+
+
+def _common(p: argparse.ArgumentParser, hyp: bool = True) -> None:
+    p.add_argument("--src", required=hyp, help="tokenised source sentences, one per line")
+    p.add_argument("--ref", required=True, help="reference M2 file (ERRANT edit types)")
+    if hyp:
+        p.add_argument("--hyp", nargs="+", required=True, metavar="NAME=PATH",
+                       help="system outputs (text, one sentence per line, or .m2); NAME= is optional")
+    p.add_argument("--type", default="R:VERB:SVA", help="error-type selector, e.g. R:VERB:SVA, DET, 'R:VERB*', 'PREP,DET'")
+    p.add_argument("--tokenise", action="store_true", help="let ERRANT tokenise the inputs (default: pre-tokenised)")
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(prog="geclens", description="Error-type-targeted evaluation of GEC systems.")
+    ap.add_argument("--version", action="version", version=f"geclens {__version__}")
+    sp = ap.add_subparsers(dest="cmd", required=True)
+
+    p = sp.add_parser("subset", help="build type-targeted subsets and reference statistics")
+    _common(p, hyp=False)
+    p.add_argument("--out", default="subsets.json")
+
+    p = sp.add_parser("score", help="score systems (point estimates and optional bootstrap CIs)")
+    _common(p)
+    p.add_argument("--out-dir", default="geclens_out")
+    p.add_argument("--ci", type=int, default=0, metavar="B", help="bootstrap resamples for 95%% CIs (0 = off)")
+    p.add_argument("--seed", type=int, default=13)
+
+    p = sp.add_parser("compare", help="paired significance tests between systems (Holm-adjusted)")
+    _common(p)
+    p.add_argument("--measures", nargs="+", default=["type_only", "overcorrection"],
+                   choices=["overall", "type", "type_only", "detection", "success", "overcorrection", "change_rate"])
+    p.add_argument("--baseline", help="compare every system with this one (default: all pairs)")
+    p.add_argument("--shuffles", type=int, default=10000)
+    p.add_argument("--resamples", type=int, default=2000, help="paired bootstrap resamples for CIs of differences")
+    p.add_argument("--seed", type=int, default=13)
+    p.add_argument("--out-dir", default="geclens_out")
+
+    p = sp.add_parser("profile", help="system x error-type heatmap of ERRANT F0.5")
+    _common(p)
+    p.add_argument("--group", choices=["type", "category"], default="category")
+    p.add_argument("--top", type=int, default=12, help="most frequent reference types/categories to show")
+    p.add_argument("--out-dir", default="geclens_out")
+
+    p = sp.add_parser("run", help="run a full evaluation from a YAML configuration file")
+    p.add_argument("config")
+
+    a = ap.parse_args(argv)
+
+    if a.cmd == "subset":
+        sub = build_subsets(a.ref, a.type)
+        sub.to_json(a.out)
+        print(json.dumps({**sub.sizes(), "selector": sub.selector}, indent=1))
+        return 0
+
+    if a.cmd == "run":
+        from .config import run_config
+        run_config(a.config)
+        return 0
+
+    from .pipeline import score_many
+    systems = _systems(a.hyp)
+    os.makedirs(a.out_dir, exist_ok=True)
+    scores = score_many(a.src, a.ref, systems, a.type, os.path.join(a.out_dir, "m2_cache"), a.tokenise)
+
+    if a.cmd == "score":
+        cis = None
+        if a.ci:
+            from .stats import bootstrap_ci
+            cis = {n: bootstrap_ci(s, resamples=a.ci, seed=a.seed) for n, s in scores.items()}
+        table = report.rows(scores, cis=cis)
+        report.write_csv(os.path.join(a.out_dir, "scores.csv"), scores)
+        md = report.markdown_table(table)
+        with open(os.path.join(a.out_dir, "scores.md"), "w", encoding="utf-8") as fh:
+            fh.write(md)
+        for n, s in scores.items():
+            s.to_json(os.path.join(a.out_dir, f"{n}.json"))
+        print(md)
+        return 0
+
+    if a.cmd == "compare":
+        from .stats import compare_systems
+        comps = compare_systems(scores, a.measures, baseline=a.baseline, shuffles=a.shuffles,
+                                resamples=a.resamples, seed=a.seed)
+        md = report.comparisons_markdown(comps)
+        with open(os.path.join(a.out_dir, "comparisons.md"), "w", encoding="utf-8") as fh:
+            fh.write(md)
+        with open(os.path.join(a.out_dir, "comparisons.json"), "w", encoding="utf-8") as fh:
+            json.dump([c.as_dict() for c in comps], fh, indent=1)
+        print(md)
+        return 0
+
+    if a.cmd == "profile":
+        from .profile import profile_table, ranked_types
+        types = ranked_types(a.ref, a.group, a.top)
+        prof = profile_table(scores, group=a.group, types=types)
+        prof = {t: prof[t] for t in types if t in prof}
+        with open(os.path.join(a.out_dir, "profile.json"), "w", encoding="utf-8") as fh:
+            json.dump(prof, fh, indent=1)
+        report.heatmap(prof, os.path.join(a.out_dir, "profile.png"), systems=list(scores))
+        print(f"profile written to {a.out_dir}/profile.png")
+        return 0
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

geclens-1.0.0/geclens/config.py

@@ -0,0 +1,52 @@
+"""YAML configuration runner: one file describes data, systems, error types and statistics.
+
+Example::
+
+    source: data/src.txt
+    reference: data/ref.m2
+    out_dir: results
+    selectors: [R:VERB:SVA, DET, PREP]      # one evaluation per selector
+    systems:
+      baseline: outputs/baseline.txt
+      improved: outputs/improved.txt
+    statistics:
+      bootstrap_resamples: 2000
+      shuffles: 10000
+      measures: [type_only, overcorrection]
+      baseline: baseline
+      seed: 13
+    profile: {group: category, top: 12}
+
+Relative paths are resolved against the directory of the configuration file.
+"""
+from __future__ import annotations
+
+import os
+
+import yaml
+
+from .pipeline import evaluate
+
+
+def run_config(path: str) -> dict:
+    with open(path, encoding="utf-8") as fh:
+        cfg = yaml.safe_load(fh)
+    base = os.path.dirname(os.path.abspath(path))
+    res = lambda p: p if os.path.isabs(p) else os.path.join(base, p)
+    selectors = cfg.get("selectors") or [cfg.get("selector", "R:VERB:SVA")]
+    stats = cfg.get("statistics", {}) or {}
+    prof = cfg.get("profile", {}) or {}
+    out_root = res(cfg.get("out_dir", "geclens_out"))
+    systems = {k: res(v) for k, v in cfg["systems"].items()}
+    results = {}
+    for i, sel in enumerate(selectors):
+        sub = out_root if len(selectors) == 1 else os.path.join(out_root, sel.replace(":", "_").replace(",", "+").replace("*", "x"))
+        results[sel] = evaluate(
+            res(cfg["source"]), res(cfg["reference"]), systems, selector=sel, out_dir=sub,
+            ci=stats.get("bootstrap_resamples", 2000) > 0, resamples=stats.get("bootstrap_resamples", 2000),
+            compare=stats.get("compare", True), measures=stats.get("measures", ["type_only", "overcorrection"]),
+            baseline=stats.get("baseline"), shuffles=stats.get("shuffles", 10000), seed=stats.get("seed", 13),
+            profile=prof.get("enabled", True) and i == 0, profile_group=prof.get("group", "category"),
+            profile_top=prof.get("top", 12), cache_dir=os.path.join(out_root, "m2_cache"),
+            tokenise=cfg.get("tokenise", False))
+    return results