extract-cli 0.1.13__tar.gz → 0.1.14__tar.gz

{extract_cli-0.1.13 → extract_cli-0.1.14}/CHANGELOG.md

@@ -6,6 +6,19 @@ to [Semantic Versioning](https://semver.org/). Per the suite convention
 (see [`docs/INTEROP.md`](docs/INTEROP.md)), **backward-incompatible changes to
 the output schema require a major version bump**; new optional fields are minor.
 
+## [0.1.14] - 2026-05-22
+
+### Improved
+- **HTML clause detection now recognizes emphasis-marked headings.** Real HTML
+  contracts (e.g. SEC EDGAR exhibits) mark section headings with emphasis, not
+  `##`/numbers — a heading tag, `<b>`/`<strong>`/`<u>`, or **CSS**
+  (`font-weight:bold` / `text-decoration:underline`), often with a leading
+  `(g)` / `1.` token and the body run-in. The HTML reader now emits such blocks
+  as `## ` headings (splitting run-in title from body; a lone emphasized block
+  is treated as a title and left plain so numbered/ALL-CAPS sections still win).
+  On the accuracy benchmark this lifts **clause recall 0.45 → 0.86** (F1 0.62 →
+  0.93), precision still 1.00. Residual misses are compound/combined headings.
+
 ## [0.1.13] - 2026-05-22
 
 ### Added
@@ -354,6 +367,7 @@ Initial release — the open-loop front door of the contract-ops CLI suite.
   intentionally *not* governed by the output schema (the schema describes the
   full default output).
 
+[0.1.14]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.14
 [0.1.13]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.13
 [0.1.12]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.12
 [0.1.11]: https://github.com/DrBaher/extract-cli/releases/tag/v0.1.11

{extract_cli-0.1.13 → extract_cli-0.1.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: extract-cli
-Version: 0.1.13
+Version: 0.1.14
 Summary: Open-loop front door of the contract-ops CLI suite: ingest any contract (.md/.txt/.html/.docx/.pdf) and emit structured JSON.
 Project-URL: Homepage, https://cli.drbaher.com/
 Project-URL: Repository, https://github.com/DrBaher/extract-cli
@@ -269,10 +269,11 @@ Line coverage tells you the code runs; it doesn't tell you the extraction is
 | effective date | accuracy 1.00 |
 | governing law | accuracy 1.00 |
 | jurisdiction (normalized) | accuracy 1.00 |
-| clauses (recall on verified sections) | 0.45 |
+| clauses (recall on verified sections) | 0.86 |
 
-Clause recall is the honest weak spot — heading detection on dense HTML
-exhibits still misses sections. A test (`tests/test_eval.py`) gates these so
+Clause recall improved sharply once the HTML reader learned to treat
+emphasis (heading tags, <b>/<u>, CSS font-weight/underline) as section
+headings; the residual misses are compound/combined heading titles. A test (`tests/test_eval.py`) gates these so
 accuracy can't silently regress.
 
 ## Development

{extract_cli-0.1.13 → extract_cli-0.1.14}/README.md

@@ -231,10 +231,11 @@ Line coverage tells you the code runs; it doesn't tell you the extraction is
 | effective date | accuracy 1.00 |
 | governing law | accuracy 1.00 |
 | jurisdiction (normalized) | accuracy 1.00 |
-| clauses (recall on verified sections) | 0.45 |
+| clauses (recall on verified sections) | 0.86 |
 
-Clause recall is the honest weak spot — heading detection on dense HTML
-exhibits still misses sections. A test (`tests/test_eval.py`) gates these so
+Clause recall improved sharply once the HTML reader learned to treat
+emphasis (heading tags, <b>/<u>, CSS font-weight/underline) as section
+headings; the residual misses are compound/combined heading titles. A test (`tests/test_eval.py`) gates these so
 accuracy can't silently regress.
 
 ## Development

{extract_cli-0.1.13 → extract_cli-0.1.14}/extract_cli.py

@@ -43,11 +43,11 @@ import urllib.request
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple
 
-__version__ = "0.1.13"
+__version__ = "0.1.14"
 
 # Bumped independently of the package version when the *extraction logic*
 # changes in a way downstream consumers should notice. Embedded in `_meta`.
-EXTRACTOR_VERSION = "0.1.13"
+EXTRACTOR_VERSION = "0.1.14"
 
 # JSON Schema version of the output contract (docs/spec/extract-output.schema.json).
 SCHEMA_VERSION = 1
@@ -1069,11 +1069,24 @@ def _detect_format(path: Path, raw: bytes) -> str:
     return base
 
 
+def _looks_like_heading_text(s: str) -> bool:
+    """Lenient: short, few words, not a full sentence -- used to decide whether
+    an *emphasized* HTML block is a clause heading."""
+    s = s.strip().rstrip(".:;,")
+    return 2 <= len(s) <= 90 and len(s.split()) <= 10
+
+
 class _HTMLTextExtractor(html.parser.HTMLParser):
-    """Stdlib HTML -> text: drops script/style, frames block elements with blank
-    lines (so clause-heading detection still works), and unescapes entities."""
+    """Stdlib HTML -> text. Drops script/style, frames blocks with blank lines,
+    unescapes entities, and -- crucially for clause detection -- emits blocks
+    that are emphasized (a heading tag, or text wrapped in <b>/<strong>/<u>) as
+    Markdown `## headings`. Real contracts (e.g. SEC HTML exhibits) mark section
+    headings with emphasis, not `##`/numbers, so without this the cascade sees
+    only plain lines. A run-in heading (emphasized lead + body in one block) is
+    split into `## Title` + body."""
 
     _SKIP = {"script", "style", "head", "title", "meta", "link", "noscript"}
+    _EMPH = {"b", "strong", "u", "h1", "h2", "h3", "h4", "h5", "h6"}
     _BLOCK = {
         "p", "div", "br", "tr", "li", "h1", "h2", "h3", "h4", "h5", "h6",
         "section", "article", "table", "ul", "ol", "blockquote", "pre", "hr",
@@ -1082,32 +1095,94 @@ class _HTMLTextExtractor(html.parser.HTMLParser):
 
     def __init__(self) -> None:
         super().__init__(convert_charrefs=True)
-        self._parts: List[str] = []
+        self._lines: List[str] = []
+        self._runs: List[Tuple[bool, str]] = []  # (emphasized, text) for current block
         self._skip = 0
+        self._emph = 0
+        # Per-tag-name LIFO stack of "did this open tag add emphasis?", so an
+        # emphasis opened by a CSS style (not just a <b>/<u> tag) is closed by
+        # the right end tag even when many <font>/<span> nest.
+        self._emph_stack: Dict[str, List[bool]] = {}
+
+    @staticmethod
+    def _style_is_emph(attrs: Any) -> bool:
+        for name, value in attrs:
+            if name == "style" and value:
+                v = value.lower()
+                if ("font-weight:bold" in v.replace(" ", "") or "font-weight:700" in v.replace(" ", "")
+                        or "text-decoration:underline" in v.replace(" ", "")):
+                    return True
+        return False
+
+    def _flush_block(self) -> None:
+        runs, self._runs = self._runs, []
+        full = re.sub(r"\s+", " ", "".join(t for _e, t in runs)).strip()
+        if not full:
+            self._lines.append("")
+            return
+        # Standalone emphasized block (a heading tag or fully <b>/<u>/styled text).
+        if all(e for e, t in runs if t.strip()) and _looks_like_heading_text(_strip_clause_number(full)):
+            self._lines.append("## " + _strip_clause_number(full))
+            return
+        # Run-in heading: an optional leading numbering token ("(g)", "1.") then
+        # an emphasized title, then the body in the same block.
+        i, saw_emph = 0, False
+        while i < len(runs):
+            emph, txt = runs[i]
+            if not txt.strip():
+                i += 1
+            elif emph:
+                saw_emph = True
+                i += 1
+            elif not saw_emph and re.fullmatch(r"\(?[0-9A-Za-z]{1,4}\)?[.)]?", txt.strip()):
+                i += 1  # skip a clause-number/letter prefix
+            else:
+                break
+        lead = _strip_clause_number(re.sub(r"\s+", " ", "".join(t for _e, t in runs[:i])).strip())
+        rest = re.sub(r"\s+", " ", "".join(t for _e, t in runs[i:])).strip()
+        if saw_emph and lead and rest and _looks_like_heading_text(lead):
+            self._lines.append("## " + lead)
+            self._lines.append(rest)
+        else:
+            self._lines.append(full)
 
     def handle_starttag(self, tag: str, attrs: Any) -> None:
         if tag in self._SKIP:
             self._skip += 1
-        elif tag in self._BLOCK:
-            self._parts.append("\n")
+            return
+        if tag in self._BLOCK:
+            self._flush_block()
+        added = tag in self._EMPH or self._style_is_emph(attrs)
+        self._emph_stack.setdefault(tag, []).append(added)
+        if added:
+            self._emph += 1
 
     def handle_endtag(self, tag: str) -> None:
         if tag in self._SKIP and self._skip > 0:
             self._skip -= 1
-        elif tag in self._BLOCK:
-            self._parts.append("\n")
+            return
+        stack = self._emph_stack.get(tag)
+        if stack:
+            if stack.pop() and self._emph > 0:
+                self._emph -= 1
+        if tag in self._BLOCK:
+            self._flush_block()
 
     def handle_data(self, data: str) -> None:
         if self._skip == 0:
-            self._parts.append(data)
+            self._runs.append((self._emph > 0, data))
 
     def get_text(self) -> str:
-        # Strip each line; collapse runs of blank lines to a single blank line
-        # (gives ALL-CAPS / numbered headings their blank-line frame).
-        lines = [re.sub(r"[ \t]+", " ", ln).strip() for ln in "".join(self._parts).split("\n")]
+        self._flush_block()
+        # A lone emphasized heading is almost always the document title, not a
+        # section scheme -- downgrade it to plain text so the numbered/ALL-CAPS
+        # tiers can still detect the real sections (matches the >=2 threshold the
+        # other fallback tiers use).
+        if sum(1 for ln in self._lines if ln.startswith("## ")) < 2:
+            self._lines = [ln[3:] if ln.startswith("## ") else ln for ln in self._lines]
         out: List[str] = []
         blank = False
-        for ln in lines:
+        for ln in self._lines:
             if ln:
                 out.append(ln)
                 blank = False

{extract_cli-0.1.13 → extract_cli-0.1.14}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "extract-cli"
-version = "0.1.13"
+version = "0.1.14"
 description = "Open-loop front door of the contract-ops CLI suite: ingest any contract (.md/.txt/.html/.docx/.pdf) and emit structured JSON."
 readme = "README.md"
 requires-python = ">=3.9"

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/employment_docx.docx.expected.json

@@ -151,7 +151,7 @@
   ],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/heading_docx.docx.expected.json

@@ -140,7 +140,7 @@
   "amounts": [],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/lease_allcaps.txt.expected.json

@@ -146,7 +146,7 @@
   ],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/license_pdf.pdf.expected.json

@@ -146,7 +146,7 @@
   ],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/nda_h2.md.expected.json

@@ -150,7 +150,7 @@
   "amounts": [],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/numbered_docx.docx.expected.json

@@ -140,7 +140,7 @@
   "amounts": [],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/scanned.pdf.expected.json

@@ -55,7 +55,7 @@
   "amounts": [],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/services_bold.txt.expected.json

@@ -146,7 +146,7 @@
   ],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/fixtures/services_html.html.expected.json

@@ -161,7 +161,7 @@
   ],
   "signatories": [],
   "_meta": {
-    "extractor_version": "0.1.13",
+    "extractor_version": "0.1.14",
     "tiers_used": [
       "deterministic"
     ],

{extract_cli-0.1.13 → extract_cli-0.1.14}/tests/test_misc.py

@@ -327,6 +327,25 @@ def test_html_extraction() -> None:
     assert {"Payment", "Termination", "Confidentiality", "Governing Law"} <= canon
 
 
+def test_html_emphasis_headings_become_clauses() -> None:
+    """Section headings marked by emphasis (heading tag, <b>/<u>, or CSS
+    font-weight/underline) -- with or without a leading '(a)'/'1.' token -- are
+    emitted as `## ` headings and detected as clauses."""
+    html = (
+        "<html><body>"
+        "<p><b>MASTER AGREEMENT</b></p>"
+        "<p>(a) <u>Confidentiality</u>. The parties keep information secret.</p>"
+        "<p><font style=\"font-weight:bold\">Payment</font>. Fees are due monthly.</p>"
+        "<p>(c) <span style=\"text-decoration:underline\">Governing Law</span>. "
+        "Governed by the laws of the State of Delaware.</p>"
+        "</body></html>"
+    )
+    text = ex._read_html(html)
+    result = ex.build_extraction(text, html.encode(), "html", "x.html")
+    canon = {c["canonical_title"] for c in result["clauses"] if c["mapped"]}
+    assert {"Confidentiality", "Payment", "Governing Law"} <= canon
+
+
 def test_html_detected_by_content_sniff(tmp_path: Any) -> None:
     # HTML masquerading as .txt (e.g. a SEC EDGAR full submission) is sniffed.
     p = tmp_path / "exhibit.txt"