extract-cli 0.1.12__tar.gz → 0.1.13__tar.gz

{extract_cli-0.1.12 → extract_cli-0.1.13}/CHANGELOG.md

@@ -6,6 +6,27 @@ to [Semantic Versioning](https://semver.org/). Per the suite convention
 (see [`docs/INTEROP.md`](docs/INTEROP.md)), **backward-incompatible changes to
 the output schema require a major version bump**; new optional fields are minor.
 
+## [0.1.13] - 2026-05-22
+
+### Added
+- **Accuracy benchmark** (`tests/eval/`, `make eval`). Scores the deterministic
+  tier against a small corpus of real, executed SEC-EDGAR contracts with
+  hand-verified ground truth, reporting precision/recall/F1 per field — turning
+  "best-effort" into a measured number. Current: parties F1 0.96, effective
+  date / governing law / jurisdiction 1.00, clause recall 0.45 (heading
+  detection on dense HTML is the known weak spot). `tests/test_eval.py` gates it
+  so accuracy can't silently regress.
+
+### Fixed / improved (surfaced by the benchmark)
+- **Governing-law detection** now covers the common connector phrasings beyond
+  "governed by the laws of X": "governed by, **and enforced in accordance
+  with,** the laws of X", "**interpreted and enforced in accordance with** the
+  laws of X", "**construed under** the laws of X". (Benchmark: governing law
+  0.67 → 1.00.)
+- **Jurisdiction normalization** now maps **all 50 US states + DC** (plus more
+  Canadian provinces / UK nations / countries), not just a dozen. (Benchmark:
+  jurisdiction 0.67 → 1.00.)
+
 ## [0.1.12] - 2026-05-22
 
 ### Security
@@ -333,6 +354,7 @@ Initial release — the open-loop front door of the contract-ops CLI suite.
   intentionally *not* governed by the output schema (the schema describes the
   full default output).
 
+[0.1.13]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.13
 [0.1.12]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.12
 [0.1.11]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.11
 [0.1.10]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.10

{extract_cli-0.1.12 → extract_cli-0.1.13}/Makefile

@@ -40,6 +40,9 @@ coverage:
 typecheck:
 	$(PYTHON) -m mypy --strict extract_cli.py
 
+eval:
+	$(PYTHON) tests/eval/evaluate.py
+
 build: clean
 	$(PYTHON) -m build
 

{extract_cli-0.1.12 → extract_cli-0.1.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: extract-cli
-Version: 0.1.12
+Version: 0.1.13
 Summary: Open-loop front door of the contract-ops CLI suite: ingest any contract (.md/.txt/.html/.docx/.pdf) and emit structured JSON.
 Project-URL: Homepage, https://cli.drbaher.com/
 Project-URL: Repository, https://github.com/DrBaher/extract-cli
@@ -256,13 +256,33 @@ paths. Configure it once and every suite tool that adopts the same lookup gets
 LLM features for free. Without it, `--llm` just warns and returns the
 deterministic output.
 
+## Accuracy
+
+Line coverage tells you the code runs; it doesn't tell you the extraction is
+*correct*. `make eval` scores the deterministic tier against a small corpus of
+**real, executed contracts** (SEC EDGAR filings) with hand-verified ground truth
+([`tests/eval/`](tests/eval/)), reporting precision/recall per field:
+
+| Field | Score |
+|---|---|
+| parties | P 1.00 · R 0.92 · F1 0.96 |
+| effective date | accuracy 1.00 |
+| governing law | accuracy 1.00 |
+| jurisdiction (normalized) | accuracy 1.00 |
+| clauses (recall on verified sections) | 0.45 |
+
+Clause recall is the honest weak spot — heading detection on dense HTML
+exhibits still misses sections. A test (`tests/test_eval.py`) gates these so
+accuracy can't silently regress.
+
 ## Development
 
 ```bash
 make install      # editable install with the [dev] extra
 make test         # full suite
-make coverage     # suite + coverage report
+make coverage     # suite + coverage report (installs extras; fails under 100%)
 make typecheck    # mypy --strict
+make eval         # accuracy benchmark vs the labeled corpus
 make build        # wheel + sdist
 make smoke        # build, install the wheel in a clean venv, run it
 make spec-check   # assert docs/spec schema == `extract schema`

{extract_cli-0.1.12 → extract_cli-0.1.13}/README.md

@@ -218,13 +218,33 @@ paths. Configure it once and every suite tool that adopts the same lookup gets
 LLM features for free. Without it, `--llm` just warns and returns the
 deterministic output.
 
+## Accuracy
+
+Line coverage tells you the code runs; it doesn't tell you the extraction is
+*correct*. `make eval` scores the deterministic tier against a small corpus of
+**real, executed contracts** (SEC EDGAR filings) with hand-verified ground truth
+([`tests/eval/`](tests/eval/)), reporting precision/recall per field:
+
+| Field | Score |
+|---|---|
+| parties | P 1.00 · R 0.92 · F1 0.96 |
+| effective date | accuracy 1.00 |
+| governing law | accuracy 1.00 |
+| jurisdiction (normalized) | accuracy 1.00 |
+| clauses (recall on verified sections) | 0.45 |
+
+Clause recall is the honest weak spot — heading detection on dense HTML
+exhibits still misses sections. A test (`tests/test_eval.py`) gates these so
+accuracy can't silently regress.
+
 ## Development
 
 ```bash
 make install      # editable install with the [dev] extra
 make test         # full suite
-make coverage     # suite + coverage report
+make coverage     # suite + coverage report (installs extras; fails under 100%)
 make typecheck    # mypy --strict
+make eval         # accuracy benchmark vs the labeled corpus
 make build        # wheel + sdist
 make smoke        # build, install the wheel in a clean venv, run it
 make spec-check   # assert docs/spec schema == `extract schema`

{extract_cli-0.1.12 → extract_cli-0.1.13}/extract_cli.py

@@ -43,11 +43,11 @@ import urllib.request
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple
 
-__version__ = "0.1.12"
+__version__ = "0.1.13"
 
 # Bumped independently of the package version when the *extraction logic*
 # changes in a way downstream consumers should notice. Embedded in `_meta`.
-EXTRACTOR_VERSION = "0.1.12"
+EXTRACTOR_VERSION = "0.1.13"
 
 # JSON Schema version of the output contract (docs/spec/extract-output.schema.json).
 SCHEMA_VERSION = 1
@@ -616,8 +616,11 @@ _ROLE_PAREN_RE = re.compile(
 # enforces a capitalized proper noun (a global re.IGNORECASE would defeat that
 # and over-capture trailing lowercase clauses like ", without regard to ...").
 _GOV_LAW_RE = re.compile(
-    r"(?i:governed\s+by(?:\s+and\s+construed\s+in\s+accordance\s+with)?\s+"
-    r"(?:the\s+)?laws?\s+of\s+(?:the\s+)?)"
+    # Allow a short same-sentence gap between "governed by" and "laws of" so the
+    # many real connector phrasings are covered: "...and construed in accordance
+    # with...", "...and enforced in accordance with...", "the internal laws of",
+    # etc. (bounded + lazy so it stays within the clause).
+    r"(?i:(?:governed|construed|interpreted|enforced)\b[^.\n]{0,60}?\blaws?\s+of\s+(?:the\s+)?)"
     r"([A-Z][A-Za-z\.\- ]+?(?:,\s*[A-Z][A-Za-z\.\- ]+?)?)"
     r"(?=[\.,;\n)]|\s+and\b|\s+without\b|$)",
 )
@@ -889,16 +892,31 @@ def extract_signatories(text: str) -> List[JSON]:
     return out
 
 
-# Free-text jurisdiction -> a normalized ISO-ish code (best-effort, common only).
+# Free-text jurisdiction -> a normalized ISO 3166-2 / ISO 3166-1 code. All 50 US
+# states + DC, common Canadian provinces, UK nations, and frequent countries.
+_US_STATES: Dict[str, str] = {
+    "alabama": "AL", "alaska": "AK", "arizona": "AZ", "arkansas": "AR",
+    "california": "CA", "colorado": "CO", "connecticut": "CT", "delaware": "DE",
+    "florida": "FL", "georgia": "GA", "hawaii": "HI", "idaho": "ID",
+    "illinois": "IL", "indiana": "IN", "iowa": "IA", "kansas": "KS",
+    "kentucky": "KY", "louisiana": "LA", "maine": "ME", "maryland": "MD",
+    "massachusetts": "MA", "michigan": "MI", "minnesota": "MN", "mississippi": "MS",
+    "missouri": "MO", "montana": "MT", "nebraska": "NE", "nevada": "NV",
+    "new hampshire": "NH", "new jersey": "NJ", "new mexico": "NM", "new york": "NY",
+    "north carolina": "NC", "north dakota": "ND", "ohio": "OH", "oklahoma": "OK",
+    "oregon": "OR", "pennsylvania": "PA", "rhode island": "RI", "south carolina": "SC",
+    "south dakota": "SD", "tennessee": "TN", "texas": "TX", "utah": "UT",
+    "vermont": "VT", "virginia": "VA", "washington": "WA", "west virginia": "WV",
+    "wisconsin": "WI", "wyoming": "WY", "district of columbia": "DC",
+}
 _JURISDICTION_CODES: Dict[str, str] = {
-    "delaware": "US-DE", "new york": "US-NY", "california": "US-CA",
-    "texas": "US-TX", "illinois": "US-IL", "massachusetts": "US-MA",
-    "washington": "US-WA", "florida": "US-FL", "nevada": "US-NV",
-    "new jersey": "US-NJ", "pennsylvania": "US-PA", "michigan": "US-MI",
+    **{name: f"US-{code}" for name, code in _US_STATES.items()},
     "ontario": "CA-ON", "quebec": "CA-QC", "british columbia": "CA-BC",
-    "england and wales": "GB-EAW", "england": "GB-ENG", "scotland": "GB-SCT",
+    "alberta": "CA-AB", "england and wales": "GB-EAW", "england": "GB-ENG",
+    "scotland": "GB-SCT", "wales": "GB-WLS", "northern ireland": "GB-NIR",
     "united kingdom": "GB", "france": "FR", "germany": "DE", "ireland": "IE",
     "singapore": "SG", "australia": "AU", "india": "IN", "netherlands": "NL",
+    "switzerland": "CH", "japan": "JP",
 }
 
 

{extract_cli-0.1.12 → extract_cli-0.1.13}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "extract-cli"
-version = "0.1.12"
+version = "0.1.13"
 description = "Open-loop front door of the contract-ops CLI suite: ingest any contract (.md/.txt/.html/.docx/.pdf) and emit structured JSON."
 readme = "README.md"
 requires-python = ">=3.9"

extract_cli-0.1.13/tests/eval/ATTRIBUTION.md

@@ -0,0 +1,20 @@
+# Benchmark corpus — sources & licensing
+
+The accuracy benchmark (`tests/eval/`) scores extract-cli against a small set of
+**real, executed contracts** filed publicly with the U.S. Securities and
+Exchange Commission (SEC EDGAR). SEC filings are public records; these exhibits
+are reproduced here, unmodified, solely as a regression/accuracy test fixture.
+
+| File | Source (SEC EDGAR) |
+|---|---|
+| `emp_celsci.txt` | CEL-SCI Corporation — Exhibit 10(ooo), employment agreement |
+| `msa_kpmg.txt` | Blade Internet Ventures / KPMG Consulting — master services agreement |
+| `services_visteon.txt` | Visteon Corporation — salaried employee lease agreement |
+| `consulting_mtm.htm` | MTM Technologies — consulting agreement |
+| `emp_arcp.htm` | American Realty Capital Properties — employment agreement |
+| `emp_quadgraphics.htm` | Quad/Graphics, Inc. — employment agreement |
+
+Ground truth (`gold.json`) was hand-verified against each document's text — the
+parties, effective date, governing law, normalized jurisdiction, and a
+verified subset of section headings. It is intentionally independent of what the
+extractor currently produces.