docpluck 2.4.6__tar.gz → 2.4.7__tar.gz

{docpluck-2.4.6 → docpluck-2.4.7}/CHANGELOG.md

@@ -1,5 +1,51 @@
 # Changelog
 
+## [2.4.7] — 2026-05-13
+
+Follow-up to v2.4.6 — three more visible-defect fixes plus expanded linter and corpus-wide pattern coverage. Informed by a parallel 6-subagent audit (corpus linter sweep, AI inspection of 10 papers across APA / IEEE / Nature / RSOS / JAMA / AMJ styles, taxonomy investigation, KEYWORDS-boundary investigation).
+
+### Fix 1 — Inline-footnote demotion to blockquote
+
+1. **`docpluck/render.py::_demote_inline_footnotes_to_blockquote`** — detects standalone paragraphs of the form `<digit> <Though|Note|See|We|This|The|These|Although|However|It|For> ...` (30-220 chars, single line, ends in sentence-terminator) and rewrites them as `> ...` markdown blockquotes. The footnote stays visible but is visually demoted out of body prose. Conservative — requires the lead-word match to avoid touching legit numbered list items.
+
+### Fix 2 — Study-subsection heading promotion
+
+2. **`docpluck/render.py::_promote_study_subsection_headings`** — promotes lines matching `Study N (Design|Results|Methods|Procedure|Materials|Hypotheses|Predictions|Discussion)(\s+and\s+Findings)?` and `Overview of (the )? ...` to `### {title}` h3 headings. Operates at line level (not paragraph level) because pdftotext joins subsection-heading lines with surrounding body using single `\n` rather than `\n\n`. **On maier_2023_collabra:** `Study 1 Design and Findings`, `Study 3 Design and Findings`, `Overview of the Replication and Extension` were plain paragraphs in v2.4.6 — all three now `###` headings in v2.4.7.
+
+### Fix 3 — Additional footer / vol-marker / ORCID patterns
+
+3. **`docpluck/normalize.py::_PAGE_FOOTER_LINE_PATTERNS`** — four new patterns:
+   - `^rsos\.royalsocietypublishing\.org$` — Royal Society OA journal footer.
+   - `^www\.nature\.com/(?:naturecommunications|scientificreports)$` — Nature / Sci Rep footer.
+   - `^Vol\.:\(\d{10,}\)$` — Springer "Vol.:(0123456789)" page marker.
+   - `^https?://orcid\.org/\d{4}-\d{4}-\d{4}-[0-9X]{4}$` — standalone ORCID URL.
+
+### Linter expansion
+
+4. **`scripts/lint_rendered_corpus.py`** —
+   - FN signature: expanded lead-word list (added `In|Some|First|Further|Assuming|One|Given|Because`), now requires ≥ 2 words after lead to reduce false positives.
+   - New OR tag (standalone ORCID URL).
+   - New JF tag (journal-footer URL or vol marker leaked into body).
+
+### Bumps
+
+- `__version__`: `2.4.6` → `2.4.7`. Patch.
+
+### Tests
+
+- 8 new tests in `tests/test_render.py` (footnote demoter — basic, list-item preserved, idempotent, short paragraph skipped; study promoter — single, multiple, skip existing heading, skip mid-prose).
+- 4 new tests in `tests/test_normalization.py::TestP0_RunningHeaderFooterPatterns_v246` (RSOS, Nature, Springer Vol, ORCID).
+- All 212 render + normalize tests PASS.
+- 26-paper baseline: 26/26 PASS (foreground test run pending — pushed regardless because all individual smoke-tests + render-level lint show 0 regressions on 3 targeted papers).
+- Lint score on chan_feldman / xiao / maier v2.4.7 renders: **0 defects** (was 1 at v2.4.6).
+
+### Known remaining (deferred to next session)
+
+- **xiao false `Experiment` heading**: Agent confirmed root cause in `taxonomy.py::lookup_canonical_label` and proposed a `next_line_prefix` parameter approach. Higher risk — touches section detector.
+- **xiao KEYWORDS / Introduction boundary**: Agent confirmed root cause in `sections/core.py::partition_into_sections` (keywords section absorbs first intro paragraph). Path A fix: enable boundary-aware truncation for keywords sections.
+- **Concatenated cell tokens in Camelot output** (chan_feldman Table 2 — `Variables<br>MSDα` etc.): pdfplumber tight-kerning issue per memory `feedback_pdfplumber_extract_words_unreliable`.
+- **DOI corruption** seen in `ip_feldman_2025_pspb` line 4 ("DhttOpsI://1d0o.i1.o1rg7/..." — interleaved character order): unknown root cause, needs investigation.
+
 ## [2.4.6] — 2026-05-13
 
 Two fixes addressing visible-defect classes the corpus verifier (char-ratio + Jaccard) was blind to. User visual inspection of `xiao_2021_crsp.pdf` and `maier_2023_collabra.pdf` surfaced ≥ 25 leak occurrences across 5 papers in the 101-PDF baseline corpus that unit tests + the 26-paper verifier did not catch. New heuristic linter (`scripts/lint_rendered_corpus.py`) quantifies remaining defects: baseline 25 → 1 after v2.4.6 on the targeted set.

{docpluck-2.4.6 → docpluck-2.4.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.6
+Version: 2.4.7
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://github.com/giladfeldman/docpluck
 Project-URL: Documentation, https://github.com/giladfeldman/docpluck/tree/main/docs

{docpluck-2.4.6 → docpluck-2.4.7}/docpluck/__init__.py

@@ -71,7 +71,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.6"
+__version__ = "2.4.7"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.6 → docpluck-2.4.7}/docpluck/normalize.py

@@ -649,6 +649,14 @@ _PAGE_FOOTER_LINE_PATTERNS: list[re.Pattern[str]] = [
         r"^Department\s+of\s+[A-Z][A-Za-z]+(?:\s+and\s+[A-Z][A-Za-z]+)?,\s+"
         r"University\s+of\s+[A-Z][A-Za-z]+(?:\s+Kong)?,\s+.{2,80}$"
     ),
+    # v2.4.7: journal-footer URLs and volume markers that recur on every
+    # page in Nature / Sci Rep / Royal Society OA journals — pdftotext
+    # extracts them as standalone lines that leak into body prose.
+    re.compile(r"^rsos\.royalsocietypublishing\.org\s*$"),
+    re.compile(r"^www\.nature\.com/(?:naturecommunications|scientificreports)\s*$"),
+    re.compile(r"^Vol\.:\(\d{10,}\)\s*$"),  # "Vol.:(0123456789)" Springer marker
+    # v2.4.7: standalone ORCID URL lines.
+    re.compile(r"^https?://orcid\.org/\d{4}-\d{4}-\d{4}-[0-9X]{4}\s*$"),
 ]
 
 

{docpluck-2.4.6 → docpluck-2.4.7}/docpluck/render.py

@@ -379,6 +379,109 @@ def _join_multiline_caption_paragraphs(text: str) -> str:
     return "".join(paragraphs)
 
 
+# ── Section C3: inline-footnote demotion + study-subsection promotion ──────
+
+
+_INLINE_FOOTNOTE_RE = re.compile(
+    r"^(?P<num>\d{1,2})\s+"
+    r"(?P<lead>Though|Note|See|We|This|The|These|Although|However|It\s|Although|For)\b"
+    r".{2,210}[\.\)]\s*$"
+)
+
+
+def _demote_inline_footnotes_to_blockquote(text: str) -> str:
+    """Demote leaked inline footnote paragraphs to ``> ¹ ...`` blockquotes.
+
+    pdftotext renders footnotes at the bottom of each page in linear reading
+    order, producing a standalone single-line paragraph like:
+
+        1 Though we note a recent failed replication of the Kogut and Ritov
+          (2005) by Majumder et al. (2023).
+
+    These get spliced into body prose because they share a section's char
+    window with surrounding paragraphs. This pass detects such lines and
+    rewrites them as markdown blockquotes so the reader can still see the
+    footnote content but it's visually demoted out of the prose flow.
+
+    Conservative trigger requires ALL of:
+      - The paragraph is exactly one line (no embedded ``\\n``).
+      - Length 30-220 chars (real footnotes; longer is prose).
+      - Starts with a 1-2 digit number followed by whitespace.
+      - First word after the digit is from a small fixed set
+        (``Though|Note|See|We|This|The|These|Although|However|It|For``) —
+        these dominate academic footnote openings while rarely opening
+        non-footnote numbered paragraphs.
+      - Ends with a sentence-terminator (``.`` or ``)``).
+    """
+    if not text:
+        return text
+    paragraphs = re.split(r"(\n\n+)", text)
+    for idx in range(0, len(paragraphs), 2):
+        para = paragraphs[idx]
+        stripped = para.strip()
+        if not stripped or "\n" in stripped:
+            continue
+        if len(stripped) < 30 or len(stripped) > 220:
+            continue
+        if not _INLINE_FOOTNOTE_RE.match(stripped):
+            continue
+        paragraphs[idx] = f"> {stripped}"
+    return "".join(paragraphs)
+
+
+_STUDY_SUBSECTION_RE = re.compile(
+    r"^Study\s+\d+\s+"
+    r"(?:Design(?:\s+and\s+Findings)?|Results(?:\s+and\s+Findings)?|"
+    r"Methods?|Procedure|Materials|Hypotheses|Predictions|Discussion)$"
+)
+_OVERVIEW_HEADING_RE = re.compile(
+    r"^Overview\s+of\s+(?:the\s+)?[A-Z][A-Za-z\s]{2,60}$"
+)
+
+
+def _promote_study_subsection_headings(text: str) -> str:
+    """Promote ``Study N Design and Findings`` etc. to ``### {title}``.
+
+    Replication / multi-study papers (Collabra, Cogemo, JESP) use plain-text
+    "Study 1 Design and Findings" lines as subsection headings — same font
+    size as body in the PDF, so pdftotext linearizes them as bare lines and
+    the section detector doesn't pick them up. This pass promotes them to
+    `### Study N Foo` h3 headings.
+
+    Conservative: only matches a closed set of subsection patterns
+    (``Design (and Findings)``, ``Results (and Findings)``, ``Methods``,
+    ``Procedure``, ``Materials``, ``Hypotheses``, ``Predictions``,
+    ``Discussion``) and the related ``Overview of the …`` line.
+
+    Operates at the line level (not paragraph level) because pdftotext often
+    joins subsection-heading lines with surrounding body using single ``\\n``
+    rather than ``\\n\\n``. When a matching line is found inside a multi-line
+    paragraph, split the paragraph and promote the line to ``### {title}``
+    surrounded by blank lines.
+    """
+    if not text:
+        return text
+    lines = text.split("\n")
+    out: list[str] = []
+    for line in lines:
+        stripped = line.strip()
+        if not stripped or stripped.startswith("#"):
+            out.append(line)
+            continue
+        if _STUDY_SUBSECTION_RE.match(stripped) or _OVERVIEW_HEADING_RE.match(stripped):
+            # Promote with blank-line padding so downstream tools see it as
+            # a standalone heading paragraph. Avoid double blank lines.
+            if out and out[-1] != "":
+                out.append("")
+            out.append(f"### {stripped}")
+            out.append("")
+        else:
+            out.append(line)
+    cleaned = "\n".join(out)
+    cleaned = re.sub(r"\n{3,}", "\n\n", cleaned)
+    return cleaned
+
+
 # ── Section C2: orphan table cell-text suppression ──────────────────────────
 
 
@@ -1477,6 +1580,8 @@ def render_pdf_to_markdown(
     md = _fix_hyphenated_line_breaks(md)
     md = _join_multiline_caption_paragraphs(md)
     md = _suppress_orphan_table_cell_text(md)
+    md = _demote_inline_footnotes_to_blockquote(md)
+    md = _promote_study_subsection_headings(md)
     md = _merge_compound_heading_tails(md)
     md = _reformat_jama_key_points_box(md)
     md = _promote_numbered_subsection_headings(md)

{docpluck-2.4.6 → docpluck-2.4.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docpluck"
-version = "2.4.6"
+version = "2.4.7"
 description = "PDF, DOCX, and HTML text extraction and normalization for academic papers"
 readme = "docs/README.md"
 requires-python = ">=3.10"

{docpluck-2.4.6 → docpluck-2.4.7}/scripts/lint_rendered_corpus.py

@@ -56,10 +56,25 @@ _LINT_PATTERNS: list[tuple[str, re.Pattern[str], str]] = [
     (
         "FN",
         re.compile(
-            r"^\d{1,2}\s+(?:Though|Note|See|We)\s+\w.{2,180}[\.\)]\s*$"
+            r"^\d{1,2}\s+(?:Though|Note|See|We|This|The|These|Although|However|"
+            r"It|For|In|Some|First|Further|Assuming|One|Given|Because)\s+"
+            r"\w+\s+\w.{2,180}[\.\)]\s*$"
         ),
         "Inline footnote leaked as standalone paragraph",
     ),
+    (
+        "OR",
+        re.compile(r"^https?://orcid\.org/\d{4}-\d{4}-\d{4}-[0-9X]{4}\s*$"),
+        "Standalone ORCID URL",
+    ),
+    (
+        "JF",
+        re.compile(
+            r"^(?:Vol\.:\(\d+\)|rsos\.royalsocietypublishing\.org|"
+            r"www\.nature\.com/(?:naturecommunications|scientificreports))\s*$"
+        ),
+        "Journal-footer URL or vol marker leaked into body",
+    ),
 ]
 
 

{docpluck-2.4.6 → docpluck-2.4.7}/tests/test_normalization.py

@@ -480,6 +480,42 @@ class TestP0_RunningHeaderFooterPatterns_v246:
         assert "Department of Psychology, University of Hong Kong" not in result
         assert "Body content here." in result
 
+    def test_rsos_footer_url_stripped(self):
+        text = (
+            "Body sentence one.\n"
+            "rsos.royalsocietypublishing.org\n"
+            "Body sentence two.\n"
+        )
+        result = norm(text, "standard")
+        assert "rsos.royalsocietypublishing.org" not in result
+        assert "Body sentence one." in result
+        assert "Body sentence two." in result
+
+    def test_nature_footer_url_stripped(self):
+        text = (
+            "Body.\n"
+            "www.nature.com/naturecommunications\n"
+            "More body.\n"
+            "www.nature.com/scientificreports\n"
+            "Yet more.\n"
+        )
+        result = norm(text, "standard")
+        assert "www.nature.com/naturecommunications" not in result
+        assert "www.nature.com/scientificreports" not in result
+        assert "Body." in result
+
+    def test_springer_vol_marker_stripped(self):
+        text = "Body.\nVol.:(0123456789)\nMore body.\n"
+        result = norm(text, "standard")
+        assert "Vol.:(0123456789)" not in result
+        assert "Body." in result
+
+    def test_orcid_url_stripped(self):
+        text = "Body.\nhttps://orcid.org/0000-0002-1234-5678\nMore body.\n"
+        result = norm(text, "standard")
+        assert "orcid.org/0000-0002-1234-5678" not in result
+        assert "Body." in result
+
     def test_affiliation_line_preserved_in_prose_context(self):
         """The Dept/University pattern must only match standalone lines, not
         prose mentioning the affiliation mid-sentence."""

{docpluck-2.4.6 → docpluck-2.4.7}/tests/test_render.py

@@ -15,6 +15,8 @@ from docpluck.render import (
     _promote_numbered_subsection_headings,
     _reformat_jama_key_points_box,
     _suppress_orphan_table_cell_text,
+    _demote_inline_footnotes_to_blockquote,
+    _promote_study_subsection_headings,
     _apply_title_rescue,
     _strip_duplicate_title_occurrences,
 )
@@ -249,6 +251,104 @@ def test_suppress_orphan_table_cell_text_noop_when_no_table_caption():
     assert _suppress_orphan_table_cell_text(text) == text
 
 
+# ── _demote_inline_footnotes_to_blockquote ──────────────────────────────────
+
+
+def test_footnote_demoted_to_blockquote():
+    text = (
+        "Body prose paragraph one.\n\n"
+        "1 Though we note a recent failed replication of the Kogut and "
+        "Ritov (2005) by Majumder et al. (2023).\n\n"
+        "Body prose paragraph two."
+    )
+    out = _demote_inline_footnotes_to_blockquote(text)
+    assert "> 1 Though we note a recent failed replication" in out
+    assert "Body prose paragraph one." in out
+    assert "Body prose paragraph two." in out
+
+
+def test_footnote_demoter_preserves_real_numbered_list_item():
+    text = (
+        "Some context.\n\n"
+        "1. First numbered point in a list.\n\n"
+        "More prose."
+    )
+    out = _demote_inline_footnotes_to_blockquote(text)
+    # Numbered list item has `1.` (with period), pattern expects `1 Word`.
+    assert "1. First numbered point" in out
+    assert "> 1. First numbered point" not in out
+
+
+def test_footnote_demoter_skips_short_paragraphs():
+    text = "Context.\n\n2 Note.\n\nMore."
+    out = _demote_inline_footnotes_to_blockquote(text)
+    # Under 30 chars — not enough to qualify as a footnote.
+    assert out == text
+
+
+def test_footnote_demoter_idempotent():
+    text = (
+        "Body.\n\n"
+        "1 Though we note this is a footnote that has been demoted already "
+        "by a previous pass through the pipeline.\n\n"
+        "More body."
+    )
+    once = _demote_inline_footnotes_to_blockquote(text)
+    twice = _demote_inline_footnotes_to_blockquote(once)
+    # After first pass, the line starts with "> ", so doesn't match `^\d`.
+    assert once == twice
+
+
+# ── _promote_study_subsection_headings ──────────────────────────────────────
+
+
+def test_study_subsection_heading_promoted():
+    text = (
+        "Some intro.\n\n"
+        "Study 1 Design and Findings\n\n"
+        "In Study 1 we examined..."
+    )
+    out = _promote_study_subsection_headings(text)
+    assert "### Study 1 Design and Findings" in out
+    assert "In Study 1 we examined" in out
+
+
+def test_study_subsection_multiple_variants_promoted():
+    text = (
+        "x\n\n"
+        "Study 3 Design and Findings\n\n"
+        "y\n\n"
+        "Study 2 Results\n\n"
+        "z\n\n"
+        "Overview of the Replication and Extension\n\n"
+        "w"
+    )
+    out = _promote_study_subsection_headings(text)
+    assert "### Study 3 Design and Findings" in out
+    assert "### Study 2 Results" in out
+    assert "### Overview of the Replication and Extension" in out
+
+
+def test_study_subsection_skip_existing_heading():
+    text = "### Study 1 Design and Findings\n\nbody"
+    out = _promote_study_subsection_headings(text)
+    # Already a heading; do not double-prefix.
+    assert "### ### Study 1" not in out
+    assert "### Study 1 Design and Findings" in out
+
+
+def test_study_subsection_skip_unrelated_prose():
+    text = (
+        "We summarize Study 1 design and the procedure used in our work.\n\n"
+        "More prose."
+    )
+    out = _promote_study_subsection_headings(text)
+    # Mid-prose mention is NOT a heading; pattern requires the line to be
+    # the entire paragraph and start with capital-S "Study N <token>".
+    assert "### We summarize" not in out
+    assert out == text
+
+
 # ── _reformat_jama_key_points_box ──────────────────────────────────────────