docpluck 2.4.44__tar.gz → 2.4.45__tar.gz

{docpluck-2.4.44 → docpluck-2.4.45}/.claude/skills/_project/lessons.md

@@ -190,3 +190,11 @@ Plus three golden snapshot files (`tests/golden/sections/*.json`) had the versio
 **Why:** Two failure modes compounded. (1) A new normalize helper added without grepping the existing `normalize_text` S-steps duplicated S3 and, placed before it, starved it. (2) The cycle was scoped from a symptom ("35 papers show ligatures") without localizing WHICH channel was at fault — the body channel was already correct.
 
 **How to detect (next time):** Before adding any glyph/encoding helper to `normalize.py`, grep the existing `S0`-`S9` / `W0*` steps for one already handling that character class — extend/unify it rather than adding a parallel path, and never insert a new step *before* an existing one that consumes the same input. Before scoping a glyph cycle, localize the defect: grep the offending glyph's lines in a recent render and confirm whether they sit in `<td>`/`<th>`/`*Table N*`/```unstructured-table``` (table/caption/fence channels — bypass `normalize_text`) or in body prose (the S-step channel).
+
+## 2026-05-16 · Cycle 13 — a heuristic guard's value depends on the false-positive surface, which differs per call site (v2.4.45)
+
+**What:** `render.py`'s two numbered-heading promoters shared a `max_lc_run >= 5` "long lowercase-word run" prose guard. It demoted legitimate descriptive headings — jdm_.2023.16 had 19 multi-level numbered subsection headings rendered as body text, with lowercase-runs up to 12 (`3.3.2.1. The quality of planning on the previous trial moderates the effect of reflection`). The fix removed the guard ENTIRELY from `_promote_numbered_subsection_headings` but KEPT it (raised 5→8) in `_promote_numbered_section_headings`.
+
+**Why:** A lowercase-word-run count genuinely cannot distinguish a descriptive section heading from prose — both have many lowercase words. What makes a line a heading is the *number shape* + capital-start + no-terminal-punctuation + single short line. For **multi-level** dotted numbering (`N.N[.N…]`) that signature is decisive — a prose line almost never begins with a multi-level dotted number — so the lc-run guard was pure harm. For **single-level** `N.` numbering the signature is weak (a `2.` line collides with an enumerated-list item), so a prose guard there still adds value as defense-in-depth. Same guard, opposite verdicts, because the false-positive surface differs between the two call sites.
+
+**How to detect (next time):** When a heuristic guard rejects legitimate inputs, do not just retune its threshold — ask whether the guard discriminates at all at that call site. Reproduce at HEAD and measure the metric's spread on real positives (here: heading lowercase-runs ran 0-12, overlapping prose entirely → no threshold works). If a guard can't separate the classes, remove it where the *other* gates already suffice and keep it only where they don't. When a guard is removed, grep its tests — a contract test (`test_render.py::test_promote_rejects_prose_with_long_lowercase_run`) was asserting the removed behavior and had to be updated in the same cycle.

{docpluck-2.4.44 → docpluck-2.4.45}/.claude/skills/docpluck-iterate/LEARNINGS.md

@@ -521,3 +521,29 @@ Mid-run, ArticleFinder flagged (and the user confirmed as a directive) that docp
 
 ### SPINE-SKIPs
 - R3 (`/docpluck-cleanup` + `/docpluck-review`) — SKIPPED. Cycle 12 is one normalize helper (explicit table over a 7-codepoint block) + S3 unified to call it + 2 bypass-channel call sites; 26/26 baseline + AI verifier confirm no regression. Same shape as cycles 2/4/6/7.
+
+---
+
+## Run: 2026-05-16 (run 4, fix-and-continue) · Cycles: cycle-12 rework, tests-regen, cycle 13
+
+This run executed `docs/HANDOFF_2026-05-16_iterate_run_4_fix_and_continue.md`'s three jobs. Cycle-12 rework + tests-regen + cycle 13 below; the article-finder AI-gold integration (JOB 2) is tracked in the run-meta.
+
+### tests-regen (commit `c831e28`, no version bump)
+- 15 pre-existing pytest failures triaged. 12 `test_extract_pdf_byte_identical` snapshots + 2 `test_sections_golden` goldens = environmental drift (local pdftotext re-wraps lines differently than the build that captured the snapshots; `extract_pdf` is a pure pdftotext passthrough). Regenerated; the 26-paper baseline is the real extraction-quality gate and stays green.
+- **The 15th, `test_request_09`, is NOT snapshot drift** — it is a real COL-class column-interleave defect: the numbered RSOS bibliography renders as `References\n1. 2. 3. ... 16.\n\nThaler RH...` (the number column split from the entry text). Left red and documented as the escalated COL defect class. Lesson: when a handoff lumps failures as "all snapshot drift," still inspect each — a real-defect-detecting test must never be "regenerated" away.
+
+### Cycle 13 (v2.4.45) — G5b long-descriptive numbered headings demoted
+
+### Outcome
+- **Cycle 13 shipped v2.4.45** — `render.py`'s numbered-heading promoters carried a `max_lc_run >= 5` prose guard that demoted legitimate long descriptive headings. Removed the guard entirely from `_promote_numbered_subsection_headings`; raised it `5→8` in `_promote_numbered_section_headings`. jdm_.2023.16: 19 multi-level subsection headings recovered.
+
+### Blind spots / process notes
+- **The TRIAGE estimate ("raise 5→8") was a partial fix.** Reproducing at HEAD showed jdm16 headings with `max_lc` up to 12 — a `5→8` raise would have left 7 of 19 still demoted. The lesson card `reproduce-triage-defect-at-head-before-trusting-cost-estimate` paid off again: always reproduce and measure before trusting a queue item's prescribed fix. The lc-run count genuinely cannot distinguish a 12-lowercase-word descriptive heading from prose — for multi-level dotted numbering the *number shape* is the discriminator, so the guard had to go, not just move.
+- **A guard worth keeping for one promoter, not the other.** Single-level `N.` numbers collide with enumerated lists (real false-positive risk) → keep a prose guard (raised to 8) as defense-in-depth. Multi-level `N.N[.N…]` numbers do not → the guard was pure harm. Same-named guard, opposite verdicts, because the false-positive surface differs.
+- **A contract test encoded the removed guard.** `test_render.py::test_promote_rejects_prose_with_long_lowercase_run` asserted the old behavior; updated it to assert the new contract (long descriptive titles ARE promoted) in the same cycle — per the cycle-2 `a test can encode the bug` lesson.
+
+### SPINE-SKIPs
+- R3 (`/docpluck-cleanup` + `/docpluck-review`) — SKIPPED. Cycle 13 is a guard removal + one threshold bump in two render post-processors; 26/26 baseline + heading-promotion-only diff confirm no regression. Same shape as cycles 9/11.
+
+### Process note — Codex cross-model verification has a Windows UTF-8 bug
+The `gold-generation.md` Step-4 Codex audit misreads UTF-8 gold files as mojibake on this Windows machine (`Västfjäll`→`VA<SI>stfjA<SI>ll`, `–`→`ƒ?"`), producing ~10-24 false "discrepancies" per paper. The gold files are confirmed clean UTF-8. Worked around by re-running Codex with an explicit "files are UTF-8; mojibake is your decode error, not a discrepancy" preamble. **This is article-finder's protocol to fix** — `gold-generation.md` Step 4 needs a UTF-8 read instruction for Windows. Flagged for coordination with the article-finder owner.

{docpluck-2.4.44 → docpluck-2.4.45}/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## [2.4.45] — 2026-05-16
+
+**Cycle 13 (autonomous APA-first run) — long descriptive numbered headings demoted to body text (G5b, S1).** `render.py`'s numbered-heading promoters carried a "long lowercase-word run" prose guard (`max_lc_run >= 5`) that rejected legitimate descriptive headings — e.g. `2.4.2.2. Inference of planning strategies and strategy types`, `3.3.2.1. The quality of planning on the previous trial moderates the effect of reflection`. jdm_.2023.16 alone had 19 multi-level numbered subsection headings demoted to body text.
+
+Fix (v2.4.45) — the lowercase-run guard is **removed from `_promote_numbered_subsection_headings`**: multi-level dotted numbering at line-start is itself a strong section-heading signal (combined with capital-started title + no terminal sentence punctuation + single ≤80-char line), and descriptive subsection titles legitimately run to many lowercase words, so the guard could not distinguish a real heading from prose and only mis-rejected headings. For `_promote_numbered_section_headings` (single-level `N.`, which genuinely collides with enumerated lists) the guard is **kept but raised `5 → 8`** — single-level promotion still has its document-numbering-range / uniqueness / list-adjacency gates as defense in depth.
+
+jdm_.2023.16: 19 previously-demoted multi-level headings now render as `###`; the v2.4.44→v2.4.45 diff is heading-promotion only (0 text loss, 0 hallucination). 26/26 baseline PASS. New real-PDF + contract tests in `tests/test_numbered_heading_promotion_real_pdf.py` and `tests/test_render.py`.
+
+~11 APA papers still FAIL Phase-5d verification; the autonomous run continues.
+
 ## [2.4.44] — 2026-05-16
 
 **Cycle 12 (autonomous APA-first run) — Latin typographic ligatures not decomposed in the table/caption channels (GLYPH, S2).** pdftotext preserves presentation-form ligature glyphs (`ff fi fl ffi ffl ſt st`, U+FB00-FB06) verbatim, so words rendered as `confident` / `influence` / `efficient` — broken for search, word matching, and any downstream NLP. A corpus scan found the glyphs in 35 rendered papers (korbmacher 82×, jdm_.2023.16 34×, jdm_m.2022.2 8×). The body channel's `normalize.py` S3 step already expanded ligatures correctly; the leak was confined to **table cells, figure/table captions, and `unstructured-table` fenced blocks**, which bypass `normalize_text` entirely.

{docpluck-2.4.44 → docpluck-2.4.45}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.44
+Version: 2.4.45
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://github.com/giladfeldman/docpluck
 Project-URL: Documentation, https://github.com/giladfeldman/docpluck/tree/main/docs

{docpluck-2.4.44 → docpluck-2.4.45}/docpluck/__init__.py

@@ -71,7 +71,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.44"
+__version__ = "2.4.45"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.44 → docpluck-2.4.45}/docpluck/render.py

@@ -233,8 +233,12 @@ def _promote_numbered_subsection_headings(text: str) -> str:
     """Promote ``1.2 Foo``-style lines to ``### 1.2 Foo`` h3 headings.
 
     Conservative: only multi-level numbering (``N.N`` or deeper), title must
-    start with a capital letter, must not end in sentence-terminator
-    punctuation, and must not look like prose (no long lowercase-word runs).
+    start with a capital letter and must not end in sentence-terminator
+    punctuation. Multi-level dotted numbering at line-start is itself a strong
+    section-heading signal — descriptive subsection titles legitimately run to
+    many lowercase words ("3.3.2.1 The quality of planning on the previous
+    trial moderates the effect of reflection"), so a lowercase-run prose guard
+    mis-rejects real headings and is not applied here (cycle 13, G5b).
     Idempotent: re-running the pass is a no-op.
     """
     if not text:
@@ -250,17 +254,6 @@ def _promote_numbered_subsection_headings(text: str) -> str:
         if title.endswith((".", "?", "!", ":", ",", ";")):
             out.append(line)
             continue
-        tokens = title.split()
-        lc_run = max_lc_run = 0
-        for tok in tokens:
-            if tok and tok[0].islower():
-                lc_run += 1
-                max_lc_run = max(max_lc_run, lc_run)
-            else:
-                lc_run = 0
-        if max_lc_run >= 5:
-            out.append(line)
-            continue
         if out and out[-1].startswith(f"### {m.group('num')} "):
             out.append(line)
             continue
@@ -357,7 +350,7 @@ def _promote_numbered_section_headings(text: str) -> str:
                 max_lc = max(max_lc, lc_run)
             else:
                 lc_run = 0
-        if max_lc >= 5:  # prose-like run — not a heading
+        if max_lc >= 8:  # long prose-like run — not a heading (cycle 13, G5b)
             continue
         candidates.setdefault(int(m.group("num")), []).append((i, title))
     if not candidates:

{docpluck-2.4.44 → docpluck-2.4.45}/docs/TRIAGE_2026-05-14_phase_5d_gold_audit.md

@@ -271,6 +271,10 @@ New `render.py::_promote_numbered_section_headings` promotes `N. Title` → `##
 
 > **Cycle-12 rework note (run 4, 2026-05-16):** the first cycle-12 attempt added a SECOND, parallel `decompose_ligatures` call *before* the pre-existing S3 step inside `normalize_text` — it consumed every ligature before S3 ran, so S3 tracked `ligatures_expanded = 0` and broke `test_normalization.py::test_report_tracks_changes`. The rework removed the duplicate call and unified S3 to use the shared helper. Lesson: before adding a glyph-normalization helper, grep the existing `normalize_text` S-steps for one already handling that glyph class — extend/unify it, do not add a parallel path.
 
+### Cycle 13 (v2.4.45) — G5b long-descriptive-title prose guard — SHIPPED
+
+`render.py`'s numbered-heading promoters carried a `max_lc_run >= 5` "long lowercase-word run" prose guard that mis-rejected legitimate descriptive headings. Reproduced at HEAD: jdm_.2023.16 alone had **19** multi-level numbered subsection headings demoted to body text, with `max_lc` up to **12** (`3.3.2.1. The quality of planning on the previous trial moderates the effect of reflection`) — far deeper than the TRIAGE's "raise 5→8" estimate. Re-scoped: the lc-run guard is **removed entirely from `_promote_numbered_subsection_headings`** (multi-level dotted numbering + capital-start + no-terminal-punctuation + single ≤80-char line is itself a sufficient heading signature; the lc-run guard cannot distinguish a descriptive heading from prose). For `_promote_numbered_section_headings` (single-level `N.`, real list-collision risk) the guard is kept but raised `5→8`, alongside its existing numbering-range/uniqueness/list-adjacency gates. jdm16: 19 headings recovered; v2.4.44→v2.4.45 diff is heading-promotion only (0 text loss/hallucination); 26/26 baseline.
+
 ### SESSION-3 STANDING VERDICT (rule 0e-bis)
 
 The APA corpus is **NOT clean**. Cycles 8-11 shipped 4 verified incremental fixes (v2.4.40-43), each AI-gold-verified OVERALL PASS with 0 regressions. But ~12 APA papers still FAIL Phase-5d on PRE-EXISTING defects the cycles did not reach. Verifier-confirmed open punch-list:
@@ -280,7 +284,7 @@ The APA corpus is **NOT clean**. Cycles 8-11 shipped 4 verified incremental fixe
 | **TABLE structure destruction** | S0/S1 | efendic, ar_apa_011, xiao, jdm15/16, chen, maier, ip_feldman (~11) | grid lost → caption-bleed; flat number-dump; empty `<table>` shells; two tables merged; rows dropped. C3 — needs a render/structured coordination design. The single largest blocker. |
 | **G5c split-line numbered headings** | S1 | jdm_m.2022.2 (`5.3.`/`6.3.`/`7.3.` etc.) | number alone on a line, title on the next; renders as orphan bare-number + a MISLABELED generic `## Results`. cycle-3 orphan-folder multi-level analogue. |
 | **G5d named (unnumbered) heading demotion** | S1 | ar_apa_011 (`Participants`, `Overview`), efendic, chandrashekar, ip_feldman (~7) | section-partitioner work; largest false-positive surface. |
-| **G5b long-descriptive-title prose guard** | S1 | jdm16, jdm_m2, chen | `≥5-lowercase-word` guard over-rejects legit long numbered headings. |
+| ~~**G5b long-descriptive-title prose guard**~~ ✓ FIXED v2.4.45 (cycle 13) | S1 | jdm16, jdm_m2, chen | ~~`≥5-lowercase-word` guard over-rejects legit long numbered headings.~~ Subsection promoter's lc-run guard removed; single-level raised 5→8. |
 | **FIG caption double-emission + truncation** | S2 | jdm_m2, efendic, chan_feldman, ziano, jdm15/16 (~8) | caption inline + in `## Figures` block; truncated mid-word; figure data-labels as orphan body lines. |
 | **GLYPH ligature** `fi`/`fl` not decomposed | S2 | jdm_m2 (and likely many) | `confident`, `influence` — NFKC would fix; check why current NFC pass misses U+FB01/FB02. |
 | **D4 metadata residuals** | S2 | ar_apa_011 (`doi:` line), chen, efendic masthead | see D4 RESIDUALS above. |

{docpluck-2.4.44 → docpluck-2.4.45}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docpluck"
-version = "2.4.44"
+version = "2.4.45"
 description = "PDF, DOCX, and HTML text extraction and normalization for academic papers"
 readme = "docs/README.md"
 requires-python = ">=3.10"

{docpluck-2.4.44 → docpluck-2.4.45}/tests/golden/sections/apa_multi_study_pdf.json

@@ -6,7 +6,7 @@
       "label": "abstract",
       "canonical_label": "abstract",
       "char_start": 0,
-      "char_end": 28,
+      "char_end": 29,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -15,8 +15,8 @@
     {
       "label": "introduction",
       "canonical_label": "introduction",
-      "char_start": 28,
-      "char_end": 53,
+      "char_start": 29,
+      "char_end": 55,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -25,8 +25,8 @@
     {
       "label": "methods",
       "canonical_label": "methods",
-      "char_start": 53,
-      "char_end": 78,
+      "char_start": 55,
+      "char_end": 81,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -35,8 +35,8 @@
     {
       "label": "results",
       "canonical_label": "results",
-      "char_start": 78,
-      "char_end": 103,
+      "char_start": 81,
+      "char_end": 107,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -45,8 +45,8 @@
     {
       "label": "methods_2",
       "canonical_label": "methods",
-      "char_start": 103,
-      "char_end": 128,
+      "char_start": 107,
+      "char_end": 133,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -55,8 +55,8 @@
     {
       "label": "results_2",
       "canonical_label": "results",
-      "char_start": 128,
-      "char_end": 153,
+      "char_start": 133,
+      "char_end": 159,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -65,8 +65,8 @@
     {
       "label": "general_discussion",
       "canonical_label": "general_discussion",
-      "char_start": 153,
-      "char_end": 183,
+      "char_start": 159,
+      "char_end": 190,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -75,8 +75,8 @@
     {
       "label": "references",
       "canonical_label": "references",
-      "char_start": 183,
-      "char_end": 213,
+      "char_start": 190,
+      "char_end": 220,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",

{docpluck-2.4.44 → docpluck-2.4.45}/tests/golden/sections/apa_single_study_pdf.json

@@ -6,7 +6,7 @@
       "label": "abstract",
       "canonical_label": "abstract",
       "char_start": 0,
-      "char_end": 36,
+      "char_end": 37,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -15,8 +15,8 @@
     {
       "label": "introduction",
       "canonical_label": "introduction",
-      "char_start": 36,
-      "char_end": 61,
+      "char_start": 37,
+      "char_end": 63,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -25,8 +25,8 @@
     {
       "label": "methods",
       "canonical_label": "methods",
-      "char_start": 61,
-      "char_end": 84,
+      "char_start": 63,
+      "char_end": 87,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -35,8 +35,8 @@
     {
       "label": "results",
       "canonical_label": "results",
-      "char_start": 84,
-      "char_end": 108,
+      "char_start": 87,
+      "char_end": 112,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -45,8 +45,8 @@
     {
       "label": "discussion",
       "canonical_label": "discussion",
-      "char_start": 108,
-      "char_end": 133,
+      "char_start": 112,
+      "char_end": 138,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",
@@ -55,8 +55,8 @@
     {
       "label": "references",
       "canonical_label": "references",
-      "char_start": 133,
-      "char_end": 163,
+      "char_start": 138,
+      "char_end": 168,
       "pages": [],
       "confidence": "high",
       "detected_via": "heading_match",