docpluck 2.4.4__tar.gz → 2.4.6__tar.gz

{docpluck-2.4.4 → docpluck-2.4.6}/.claude/skills/docpluck-qa/SKILL.md

@@ -250,6 +250,81 @@ python scripts/verify_corpus.py --paper efendic_2022_affect --diff
 Skips cleanly when the spike `outputs[-new]/` directories aren't on
 disk (fresh checkouts).
 
+### 7c. Visible-Defect Heuristic Linter (v2.4.6+, CRITICAL)
+
+`verify_corpus.py` measures char-ratio + Jaccard against a baseline — it is
+**blind to visible defects** that the baseline itself contains. After the
+2026-05-13 audit (xiao_2021_crsp, maier_2023_collabra) the user identified
+five visible defect classes that the corpus verifier missed entirely:
+
+| Defect | Signature regex (on rendered .md, per-line) | Tag |
+|---|---|---|
+| Running header `Q. XIAO ET AL.` style | `^[A-Z]\.(?:\s*[A-Z]\.?)?\s+[A-Z]{2,}\s+ET\s+AL\.?$` | RH |
+| Contact / corresponding-author footer | `^CONTACT\s+[A-Z]\w+(?:\s+[A-Z]\w+)+\s+\S+@` | CT |
+| Prefixed contribution / corresponding footnote | `^[a-c]\s+(?:Contributed\s+equally\|Corresponding\s+Author)\b` | CB |
+| Standalone Dept/University affiliation | `^Department\s+of\s+[A-Z]\w+,\s+University\s+of\s+\w+` | AF |
+| Inline footnote leaked into prose | `^\d+\s+(?:Though\|Note\|See\|We)\s+\w` (per-line, ≤ 200 chars) | FN |
+
+Run:
+```bash
+python scripts/lint_rendered_corpus.py tmp/renders_*/*.md
+```
+
+Any match is a FAIL — the rendered .md contains a defect class that should
+have been stripped upstream. Cite the file + line + tag in the QA report.
+
+Note: these patterns target the **rendered output**, not pdftotext. They
+backstop normalize.py + render.py — if a pattern leaks past upstream
+filters and into the .md, the linter catches it.
+
+### 7d. AI Inspection of Rendered Output (v2.4.6+, RECOMMENDED)
+
+For 2-5 representative papers per render change, dispatch a Claude subagent
+(via Task or Agent tool) that:
+
+1. Reads `tmp/<paper>.md` (the rendered output).
+2. Reads the source PDF (via Read tool with `pages=1-5` for the first 5 pages).
+3. Scores each .md section for fidelity:
+   - **Text coverage**: any PDF paragraph missing from the .md?
+   - **Section boundaries**: does the heading match the content below it?
+   - **Mid-prose leaks**: any running-header / footer / footnote text infused?
+   - **False headings**: any `## ...` / `### ...` that isn't actually a section?
+
+Output a per-paper defect list. **Default papers:** `xiao_2021_crsp`,
+`maier_2023_collabra`, `chan_feldman_2025_cogemo`, `efendic_2022_affect`,
+`ip_feldman_2025_pspb` — these collectively exercise APA stats tables,
+Collabra footnotes, T&F contact-line footers, sequential page numbers,
+and replication-report subsections.
+
+This check exists because **char-ratio + Jaccard are blind to "right words
+in wrong order under wrong heading"** (see CLAUDE.md "Iteration discipline").
+Run it after every render change before declaring the iteration done.
+
+### 7e. Text-Coverage Baseline (v2.4.6+, CRITICAL)
+
+Catches the silent-text-loss defect class (rendered .md drops a body
+paragraph that was in the PDF):
+
+```bash
+python -c "
+from pathlib import Path
+from docpluck.extract import extract_pdf
+from docpluck.render import render_pdf_to_markdown
+for pdf_path in Path('../PDFextractor/test-pdfs').glob('**/*.pdf'):
+    pdf = pdf_path.read_bytes()
+    raw, _ = extract_pdf(pdf)
+    md = render_pdf_to_markdown(pdf)
+    ratio = len(md) / max(len(raw), 1)
+    if ratio < 0.85:
+        print(f'COVERAGE FAIL {pdf_path.name}: {ratio:.2f}')
+    elif ratio > 2.0:
+        print(f'COVERAGE WARN {pdf_path.name}: {ratio:.2f} (suspicious bloat)')
+"
+```
+
+**Threshold:** rendered .md length ≥ 0.85 × pdftotext raw length. Below
+that, body content has been dropped somewhere in the pipeline.
+
 ### 8. Service Health Endpoint
 ```bash
 curl -s http://localhost:6117/health

{docpluck-2.4.4 → docpluck-2.4.6}/CHANGELOG.md

@@ -1,5 +1,68 @@
 # Changelog
 
+## [2.4.6] — 2026-05-13
+
+Two fixes addressing visible-defect classes the corpus verifier (char-ratio + Jaccard) was blind to. User visual inspection of `xiao_2021_crsp.pdf` and `maier_2023_collabra.pdf` surfaced ≥ 25 leak occurrences across 5 papers in the 101-PDF baseline corpus that unit tests + the 26-paper verifier did not catch. New heuristic linter (`scripts/lint_rendered_corpus.py`) quantifies remaining defects: baseline 25 → 1 after v2.4.6 on the targeted set.
+
+### Fix 1 — Orphan table cell-text suppression
+
+1. **`docpluck/render.py::_suppress_orphan_table_cell_text`** — new post-processor inserted between `_join_multiline_caption_paragraphs` and `_merge_compound_heading_tails`. Detects single-line `Table N. <caption>` paragraphs (plain, not already italicized — the italic `*Table N. ...*` is the v2.4.2 caption-only emission and never has orphan rows) followed by ≥ 3 consecutive paragraphs matching `_is_orphan_cell_paragraph` (≤ 200 chars, no markdown/HTML/list markers, low stopword density, not multi-sentence prose). When detected: italicizes the caption and drops the orphan paragraphs. Conservative: stops at the first non-orphan paragraph.
+
+On `chan_feldman_2025_cogemo`: 5 of 9 captions (Tables 3, 4, 5, 6, 7) were plain `Table N.` lines followed by 3–50 lines of orphan cell rows; all now italicized with zero orphan rows.
+
+### Fix 2 — Running-header / contact-block / affiliation line patterns
+
+2. **`docpluck/normalize.py::_PAGE_FOOTER_LINE_PATTERNS`** — four new patterns:
+   - `^[A-Z]\.(?:\s*[A-Z]\.?)?\s+[A-Z]{2,}\s+ET\s+AL\.?$` — `Q. XIAO ET AL.` / `Q.M. SMITH ET AL` running headers (all-caps surname required to avoid stripping legit `Q. Xiao et al.` references in prose).
+   - `^CONTACT\s+[A-Z]\w+(?:\s+[A-Z]\w+)+\s+\S+@\S+.*$` — Taylor & Francis (CRSP, etc.) `CONTACT <Name> <email>` page-footer.
+   - `^[a-c]\s+(?:Contributed\s+equally|Corresponding\s+Author)\b.*$` — Collabra-style prefixed contribution / corresponding-author footnotes.
+   - `^Department\s+of\s+[A-Z]\w+(?:\s+and\s+\w+)?,\s+University\s+of\s+\w+(?:\s+Kong)?,\s+.{2,80}$` — standalone Dept/University affiliation lines (must be standalone — prose mentioning the affiliation mid-sentence stays).
+
+On `xiao_2021_crsp`: 18 `Q. XIAO ET AL.` standalone leaks → 0 (one residual is folded inside a figure caption, not at line start). On `maier_2023_collabra`: 3 contact/corresponding leaks → 0.
+
+### New: heuristic linter
+
+3. **`scripts/lint_rendered_corpus.py`** — greps rendered `.md` for 5 leak signatures (RH, CT, CB, AF, FN). Run `python scripts/lint_rendered_corpus.py tmp/renders_v2.4.0/` against the 101-PDF corpus to surface visible defects char-ratio/Jaccard miss. Wired into `docpluck-qa` skill as Check 7c.
+
+### New: QA skill spec updates
+
+4. **`.claude/skills/docpluck-qa/SKILL.md`** — three new checks documented:
+   - 7c: Visible-Defect Heuristic Linter (the `lint_rendered_corpus.py` script).
+   - 7d: AI Inspection of Rendered Output (Claude subagent compares `.md` paragraph-by-paragraph against source PDF).
+   - 7e: Text-Coverage Baseline (asserts `len(rendered.md) ≥ 0.85 × len(pdftotext_raw)` to catch silent text-loss).
+
+### Bumps
+
+- `__version__`: `2.4.5` → `2.4.6`. Patch (additive normalize patterns + new render post-processor; no API surface change).
+
+### Tests
+
+- 7 new tests in `tests/test_render.py` for `_suppress_orphan_table_cell_text` (drops leaked rows, preserves prose, requires ≥ 3 orphans, skips already-italic caption, stops at next caption, idempotent, no-op when no caption).
+- 7 new tests in `tests/test_normalization.py::TestP0_RunningHeaderFooterPatterns_v246` for the new footer patterns (Q. XIAO ET AL. stripping, two-initials variant, mixed-case preservation, CONTACT footer, prefixed Contributed equally, Dept/University standalone, Dept/University prose preserved).
+
+### Known remaining defects (deferred to next iteration)
+
+- `xiao_2021_crsp`: section detector treats mid-paragraph "Experiment" as a heading. Requires context-aware suppression in `sections/taxonomy.py`.
+- `xiao_2021_crsp`: KEYWORDS section boundary not visually separated from Introduction body in render output.
+- `maier_2023_collabra`: subsection headings like "Study 1 Design and Findings" / "Study 3 Design and Findings" remain plain paragraphs — need a subsection-pattern detector in `sections/`.
+- `maier_2023_collabra`: inline footnote leak (`1 Though we note ...`) — F1 footnote post-processing pass needed.
+
+## [2.4.5] — 2026-05-13
+
+Continuation of v2.4.3's 4-digit page-number strip. v2.4.3 required the same 4-digit value to recur ≥ 3 times to strip — but continuous-pagination journals (PSPB, Psychological Science) use *sequential* page numbers per page (1174, 1175, 1177, 1179, ...) where each value is different. The v2.4.3 rule missed them entirely.
+
+### Fix
+
+1. **`docpluck/normalize.py::normalize_text` S9** — widened 4-digit page-number strip with a second pattern: when ≥ 3 distinct standalone 4-digit values cluster within a 50-page range AND have mean inter-value gap ≤ 3, treat them all as continuous-pagination page numbers and strip. The conservative gates (max-min spread, mean diff) protect against table-cell values which would have larger spreads and irregular gaps. Verified end-to-end on `efendic_2022_affect.md` — page numbers 1174, 1175, 1177, 1179, 1181, 1183, 1184 now all stripped. `NORMALIZATION_VERSION`: `1.8.2` → `1.8.3`.
+
+### Bumps
+
+- `__version__`: `2.4.4` → `2.4.5`. Patch.
+
+### Tests
+
+2 new tests in `tests/test_normalization.py` (sequential page-number stripping, unrelated 4-digit value preservation).
+
 ## [2.4.4] — 2026-05-13
 
 Bug fix on v2.4.3's caption-trim feature + extension to a second chart-data signature.

{docpluck-2.4.4 → docpluck-2.4.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.4
+Version: 2.4.6
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://github.com/giladfeldman/docpluck
 Project-URL: Documentation, https://github.com/giladfeldman/docpluck/tree/main/docs

{docpluck-2.4.4 → docpluck-2.4.6}/docpluck/__init__.py

@@ -71,7 +71,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.4"
+__version__ = "2.4.6"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.4 → docpluck-2.4.6}/docpluck/normalize.py

@@ -22,7 +22,7 @@ class NormalizationLevel(str, Enum):
     academic = "academic"
 
 
-NORMALIZATION_VERSION = "1.8.2"
+NORMALIZATION_VERSION = "1.8.3"
 
 
 # ── Request 9 (Scimeto, 2026-04-27): Reference-list normalization ──────────
@@ -617,6 +617,38 @@ _PAGE_FOOTER_LINE_PATTERNS: list[re.Pattern[str]] = [
     # Running-header lines with "| <page>" or "<page> Author et al.".
     re.compile(r"^\S(?:[^|\n]{2,80})\|\s*\d{1,4}\s*$"),
     re.compile(r"^\d{1,4}\s+[A-ZÀ-ÿ][^\n]{1,60}\s+et al\.?\s*$"),
+    # v2.4.6: "Q. XIAO ET AL." style running header — surname journal abbrev
+    # used by CRSP, JESP, and many other 2-column journals. Accepts:
+    #   "Q. XIAO ET AL."         single initial + surname
+    #   "Q.M. XIAO ET AL."       two initials with internal period
+    #   "Q. M. XIAO ET AL"       two initials with space (no trailing dot)
+    # All-caps surname required (lowercase letters appear in regular prose
+    # like "Most participants in the experimental condition were …").
+    re.compile(
+        r"^[A-Z]\.(?:\s*[A-Z]\.?)?\s+[A-Z]{2,}\s+ET\s+AL\.?\s*$"
+    ),
+    # v2.4.6: contact-line footer used by Taylor & Francis (CRSP, etc.):
+    #   "CONTACT Gilad Feldman gfeldman@hku.hk; giladfel@gmail.com …"
+    # The `CONTACT` keyword + name + email is distinctive enough to anchor
+    # safely. Optional trailing affiliation / region tokens.
+    re.compile(
+        r"^CONTACT\s+[A-Z][\w'’-]+(?:\s+[A-Z][\w'’-]+)+\s+\S+@\S+.*$"
+    ),
+    # v2.4.6: prefixed author-contribution / corresponding-author footnotes
+    # used by Collabra, eLife, PLOS, etc.:
+    #   "a Contributed equally, joint first author"
+    #   "b Contributed equally, joint first author"
+    #   "c Corresponding Author: <name>, <affiliation>"
+    re.compile(
+        r"^[a-z]\s+(?:Contributed\s+equally|Corresponding\s+Author)\b.*$"
+    ),
+    # v2.4.6: standalone affiliation lines that recur on bottom of every
+    # page in 2-column journals — "Department of <field>, University of
+    # <place>, <region>".
+    re.compile(
+        r"^Department\s+of\s+[A-Z][A-Za-z]+(?:\s+and\s+[A-Z][A-Za-z]+)?,\s+"
+        r"University\s+of\s+[A-Z][A-Za-z]+(?:\s+Kong)?,\s+.{2,80}$"
+    ),
 ]
 
 
@@ -1006,27 +1038,45 @@ def normalize_text(
         t = "\n".join(lines)
     # Strip standalone page numbers — 1-3 digit unconditionally.
     t = re.sub(r"^\s*\d{1,3}\s*$", "", t, flags=re.MULTILINE)
-    # v2.4.3: 4-digit page numbers (continuous-pagination journals like PSPB
-    # where volume runs page numbers into the 1000s). Strip when ALL of:
-    #   1. The line is exactly 4 ASCII digits.
-    #   2. The value falls in the plausible page-number range 1000–9999
-    #      (avoids stripping a stray 4-digit year-on-its-own-line).
-    #   3. The SAME value recurs ≥3 times in the document (page numbers
-    #      repeat once per physical page, so this is conservative; a
-    #      duplicate-by-coincidence table-cell value would need to be the
-    #      same number 3 times, which is rare).
-    # The conservative threshold protects table data where a 4-digit value
-    # might legitimately appear on its own line (single-value-per-line
-    # column layouts).
+    # v2.4.3/v2.4.5: 4-digit page numbers (continuous-pagination journals like
+    # PSPB where volume runs page numbers into the 1000s, e.g.
+    # ``efendic_2022_affect`` with pages 1174-1185). Two patterns fire:
+    #
+    #   (A) RECURRING (v2.4.3) — same value appears ≥3 times. Catches PDFs
+    #       where every page repeats the same volume number on its own line
+    #       (rare for true page numbers, but happens for volume markers).
+    #
+    #   (B) SEQUENTIAL (v2.4.5) — ≥3 distinct standalone 4-digit values in
+    #       the doc AND they cluster within a 50-page range (max - min ≤ 50)
+    #       AND the average per-page gap is small (mean diff ≤ 3). This is
+    #       the canonical continuous-pagination signature: page numbers
+    #       monotonically increasing across the article. The conservative
+    #       gates protect table cells (where 4-digit values would have
+    #       larger spreads and irregular gaps).
     four_digit_counts: dict[str, int] = {}
     for ln in t.split("\n"):
         s = ln.strip()
         if len(s) == 4 and s.isascii() and s.isdigit() and 1000 <= int(s) <= 9999:
             four_digit_counts[s] = four_digit_counts.get(s, 0) + 1
-    recurring_4d = {s for s, c in four_digit_counts.items() if c >= 3}
-    if recurring_4d:
+
+    # Pattern A: same value recurs ≥3 times.
+    strip_set: set[str] = {s for s, c in four_digit_counts.items() if c >= 3}
+
+    # Pattern B: ≥3 distinct values clustered tightly together.
+    if len(four_digit_counts) >= 3:
+        values = sorted(int(s) for s in four_digit_counts.keys())
+        spread = values[-1] - values[0]
+        if spread <= 50:
+            # Compute mean of consecutive diffs.
+            diffs = [values[i + 1] - values[i] for i in range(len(values) - 1)]
+            mean_diff = sum(diffs) / len(diffs)
+            if mean_diff <= 3.0:
+                # All values in the cluster are page numbers.
+                strip_set.update(str(v) for v in values)
+
+    if strip_set:
         t = "\n".join(
-            "" if ln.strip() in recurring_4d else ln
+            "" if ln.strip() in strip_set else ln
             for ln in t.split("\n")
         )
     report._track("S9_header_footer_removal", before, t, "headers_removed")

{docpluck-2.4.4 → docpluck-2.4.6}/docpluck/render.py

@@ -379,6 +379,130 @@ def _join_multiline_caption_paragraphs(text: str) -> str:
     return "".join(paragraphs)
 
 
+# ── Section C2: orphan table cell-text suppression ──────────────────────────
+
+
+_ORPHAN_TABLE_CAPTION_RE = re.compile(
+    r"^Table\s+(\d+)[.:]\s+(.{3,}?)$"
+)
+_ORPHAN_CELL_STOPWORDS = (
+    " the ", " of ", " and ", " in ", " to ", " for ", " with ", " that ",
+    " this ", " was ", " were ", " are ", " is ", " have ", " has ",
+    " from ", " on ", " by ", " an ", " a ",
+)
+
+
+def _is_orphan_cell_paragraph(p: str) -> bool:
+    """Return True iff ``p`` looks like a leaked table cell row, not prose.
+
+    Conservative heuristic, used only inside the table-cell-text suppressor:
+    - Total length ≤ 200 chars (cell content with quoted instruction text or
+      concatenated column headers can run 100-200 chars on a single pdftotext
+      line; longer than that is almost certainly prose).
+    - Not a heading, caption, HTML block, or list marker.
+    - Stopword-density and sentence-structure check rule out short prose.
+    """
+    if not p:
+        return False
+    if len(p) > 200:
+        return False
+    if p.startswith(("#", "*Table", "*Figure", "<table", "</table", "<thead", "<tbody", "<tr", "<td", "<th", ">")):
+        return False
+    if re.match(r"^(?:Table|Figure)\s+\d", p):
+        return False
+    if re.match(r"^[*+\-]\s", p) or re.match(r"^\d+\.\s+\w+", p):
+        # Markdown list / numbered list — not a cell row.
+        # (Numbered ranks like "1. Degree of apology" inside cells can match,
+        # but those are typically inside <td> tags, not standalone paragraphs.)
+        return False
+    if p.startswith("Note") and (":" in p[:8] or "." in p[:8]):
+        return False
+    lower = " " + p.lower() + " "
+    stopword_hits = sum(lower.count(sw) for sw in _ORPHAN_CELL_STOPWORDS)
+    # Above 90 chars, prose density must be very low (cells with quoted
+    # instruction text or column-header concatenations have ≤ 3 stopwords).
+    if len(p) > 90 and stopword_hits >= 4:
+        return False
+    if len(p) <= 90 and stopword_hits >= 3:
+        return False
+    # Multi-sentence content is prose, not a cell row.
+    if p.count(". ") >= 2:
+        return False
+    # Single long sentence ending in `.` (not `."` — cells often end in `"`)
+    # is prose.
+    if p.endswith(".") and not p.endswith(('."', '.")')) and len(p) > 70 and " " in p:
+        return False
+    return True
+
+
+def _suppress_orphan_table_cell_text(text: str) -> str:
+    """Suppress orphan cell-row text leaks after a plain-text Table caption.
+
+    When Camelot does not register a table on a page but pdftotext linearized
+    the cell content into the section body, the rendered markdown contains:
+
+        Table 5. Comparison of target article versus replication.
+
+        Target article
+
+        Replication
+
+        Study design
+
+        Sample characteristics
+
+    These short orphan paragraphs are leaked cell content with no structural
+    value in the rendered view (the user is told to consult the Raw view).
+    This pass:
+      1. Detects single-line ``Table N. <caption>`` paragraphs (plain, not
+         already italicized — the italic ``*Table N. ...*`` form is the
+         v2.4.2 caption-only emission and never has orphan rows).
+      2. Scans forward; if 3+ consecutive paragraphs match
+         :func:`_is_orphan_cell_paragraph`, italicizes the caption and drops
+         the orphan paragraphs.
+
+    Conservative: only fires after a ``Table N.`` caption and only when the
+    orphan run is at least 3 paragraphs long. Stops at the first non-orphan
+    paragraph (normal prose, another caption, or a heading).
+    """
+    if not text or "Table" not in text:
+        return text
+    paragraphs = re.split(r"\n\n+", text)
+    out: list[str] = []
+    i = 0
+    while i < len(paragraphs):
+        para = paragraphs[i]
+        para_stripped = para.strip()
+        # Caption must be a single line (no embedded newlines after strip).
+        if (
+            para_stripped
+            and "\n" not in para_stripped
+            and not para_stripped.startswith("*")
+            and _ORPHAN_TABLE_CAPTION_RE.match(para_stripped)
+        ):
+            j = i + 1
+            orphans: list[int] = []
+            while j < len(paragraphs):
+                p = paragraphs[j].strip()
+                if not p:
+                    j += 1
+                    continue
+                if _is_orphan_cell_paragraph(p):
+                    orphans.append(j)
+                    j += 1
+                    continue
+                break
+            if len(orphans) >= 3:
+                # Italicize the caption (matches v2.4.2 no-cells caption style)
+                # and drop the orphan paragraphs.
+                out.append(f"*{para_stripped}*")
+                i = j
+                continue
+        out.append(para)
+        i += 1
+    return "\n\n".join(out)
+
+
 # ── Section D: JAMA Key Points sidebar reformat ─────────────────────────────
 
 
@@ -1352,6 +1476,7 @@ def render_pdf_to_markdown(
     md = _dedupe_h2_sections(md)
     md = _fix_hyphenated_line_breaks(md)
     md = _join_multiline_caption_paragraphs(md)
+    md = _suppress_orphan_table_cell_text(md)
     md = _merge_compound_heading_tails(md)
     md = _reformat_jama_key_points_box(md)
     md = _promote_numbered_subsection_headings(md)