docpluck 2.4.3__tar.gz → 2.4.4__tar.gz

{docpluck-2.4.3 → docpluck-2.4.4}/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## [2.4.4] — 2026-05-13
+
+Bug fix on v2.4.3's caption-trim feature + extension to a second chart-data signature.
+
+### Bug fix
+
+1. **`docpluck/extract_structured.py::_extract_caption_text`** — v2.4.3's `_trim_caption_at_chart_data` was added to `docpluck/figures/detect.py::_full_caption_text`, but the live render pipeline never calls that function — figure captions are built in `extract_structured.py::_extract_caption_text` (which `_figure_from_caption` calls). v2.4.3's caption-trim was therefore a no-op on real renders despite its tests passing in isolation. v2.4.4 applies the trim to `_extract_caption_text` for `kind == "figure"` captions, so the trim actually fires during `render_pdf_to_markdown(pdf_bytes)`. Verified by manual render of `jama_open_6` (caption 400 chars → 47 chars) and `jama_open_3` (405 → 208 chars).
+
+### Enhancement
+
+2. **`docpluck/extract_structured.py::_trim_caption_at_chart_data`** — extended with a second chart-data signature: a run of 5+ short (1–4 digit) numeric tokens separated only by whitespace. Catches axis-tick label sequences (``0 5 10 15 20``) and stacked column values (``340 321 280 5 270``) that the 6-digit-run rule didn't see on charts with small-magnitude data. The two signatures are evaluated jointly; the earlier match in the caption wins so the caption is trimmed at the start of the chart data, not partway through it. Same conservative gates as before (caption ≥ 150 chars, surviving text ≥ 40 chars). Affects most JAMA Network Open Kaplan-Meier and Sci Rep / BMC clinical-trial papers — caption length drops from 400-char hard cap to ~150 chars of real prose.
+
+### Bumps
+
+- `__version__`: `2.4.3` → `2.4.4`. Patch — figure-caption truncation is now real and broader.
+
+### Tests
+
+3 new tests in `tests/test_figure_detect.py` (tick-run truncation, prose-with-inline-numbers no-op, earlier-of-two-signatures priority).
+
 ## [2.4.3] — 2026-05-13
 
 Same-day follow-up. Two preventative improvements aimed at quality issues that didn't trip the verifier tags but were visible in rendered output:

{docpluck-2.4.3 → docpluck-2.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.3
+Version: 2.4.4
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://github.com/giladfeldman/docpluck
 Project-URL: Documentation, https://github.com/giladfeldman/docpluck/tree/main/docs

{docpluck-2.4.3 → docpluck-2.4.4}/docpluck/__init__.py

@@ -71,7 +71,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.3"
+__version__ = "2.4.4"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.3 → docpluck-2.4.4}/docpluck/extract_structured.py

@@ -332,11 +332,57 @@ def _extract_caption_text(
     # Re-prefix the label if stripping ate it.
     if cap.label and not snippet.startswith(cap.label):
         snippet = f"{cap.label}. {snippet}".strip()
+    # v2.4.4: trim chart-data appendage from figure captions (axis-tick
+    # sequences, raw bar-chart values pdftotext joined inline into the
+    # caption paragraph). For tables the appendage is usually the next-
+    # row continuation so skip — the caption hard-cap at 400 below
+    # bounds it.
+    if cap.kind == "figure":
+        snippet = _trim_caption_at_chart_data(snippet)
     if len(snippet) > 400:
         snippet = snippet[:400].rsplit(" ", 1)[0] + "…"
     return snippet
 
 
+# v2.4.4: shared chart-data trim, duplicated logic from
+# ``docpluck.figures.detect._trim_caption_at_chart_data`` so this module
+# doesn't import from ``figures.detect`` (which has its own layout-channel
+# dependencies). Two signatures of pdftotext-joined chart data:
+#   1. Run of 6+ consecutive digits — flowchart counts, row IDs.
+#   2. Run of 5+ short (1–4 digit) numeric tokens separated only by
+#      whitespace — axis-tick label sequences.
+_CHART_DATA_DIGIT_RUN_RE_STRUCT = re.compile(r"\b\d{6,}\b")
+_CHART_DATA_TICK_RUN_RE_STRUCT = re.compile(r"(?:\b\d{1,4}\b[ \t]+){5,}")
+
+
+def _trim_caption_at_chart_data(caption: str) -> str:
+    """Truncate a caption when it transitions from prose to chart-data.
+
+    Conservative: only fires when caption ≥ 150 chars AND the surviving
+    trimmed text is ≥ 40 chars. The two regex signatures catch
+    complementary chart-data patterns (large counts and small axis-tick
+    sequences); the earlier match wins.
+    """
+    if not caption or len(caption) < 150:
+        return caption
+    candidates: list[int] = []
+    m1 = _CHART_DATA_DIGIT_RUN_RE_STRUCT.search(caption)
+    if m1 is not None:
+        candidates.append(m1.start())
+    m2 = _CHART_DATA_TICK_RUN_RE_STRUCT.search(caption)
+    if m2 is not None:
+        candidates.append(m2.start())
+    if not candidates:
+        return caption
+    cut = min(candidates)
+    while cut > 0 and not caption[cut - 1].isspace():
+        cut -= 1
+    trimmed = caption[:cut].rstrip(" ,;:")
+    if len(trimmed) < 40:
+        return caption
+    return trimmed
+
+
 def _isolated_table_from_caption(
     cap: CaptionMatch,
     raw_text: str,

{docpluck-2.4.3 → docpluck-2.4.4}/docpluck/figures/detect.py

@@ -146,6 +146,14 @@ def _full_caption_text(raw_text: str, cap: CaptionMatch) -> str:
 # that pdftotext joined chart data (raw bar-chart values, participant counts,
 # row IDs) into the caption.
 _CHART_DATA_DIGIT_RUN_RE = re.compile(r"\b\d{6,}\b")
+# A run of 5+ short numeric tokens (1–4 digits each) separated only by
+# whitespace is a v2.4.4 signal — captures axis-tick label sequences
+# (``0 5 10 15 20``) and stacked column values (``340 321 280 5 270``)
+# that the 6-digit rule misses on charts with small-magnitude data.
+# Real captions reference numbers via prose ("with n = 1234 participants",
+# "p < .001"), so digit tokens are interleaved with words rather than
+# stacked five-in-a-row.
+_CHART_DATA_TICK_RUN_RE = re.compile(r"(?:\b\d{1,4}\b[ \t]+){5,}")
 
 
 def _trim_caption_at_chart_data(caption: str) -> str:
@@ -161,21 +169,27 @@ def _trim_caption_at_chart_data(caption: str) -> str:
     where the real caption is "Flowchart of Study Sample Selection" and the
     rest is chart data values.
 
-    Strategy: locate the first run of 6+ consecutive digits (the signature
-    of chart data — counts or row IDs that no real caption sentence would
-    contain). Truncate the caption just before that run, falling back to
-    the word boundary so we don't end mid-word.
+    v2.4.4: two complementary signatures are scanned (see module-level
+    constants); the *earlier* match in the caption wins so the caption is
+    trimmed at the start of the chart data, not partway through it.
 
     Conservative: only fires when the caption is ≥ 150 chars (real short
-    captions almost never have chart-data appendage), and only when the
-    surviving trimmed caption is ≥ 40 chars after the label.
+    captions almost never have a chart-data appendage), and only when the
+    surviving trimmed caption is ≥ 40 chars (sanity check protects against
+    edge cases where the digit run lands near the label).
     """
     if not caption or len(caption) < 150:
         return caption
-    m = _CHART_DATA_DIGIT_RUN_RE.search(caption)
-    if m is None:
+    candidates: list[int] = []
+    m1 = _CHART_DATA_DIGIT_RUN_RE.search(caption)
+    if m1 is not None:
+        candidates.append(m1.start())
+    m2 = _CHART_DATA_TICK_RUN_RE.search(caption)
+    if m2 is not None:
+        candidates.append(m2.start())
+    if not candidates:
         return caption
-    cut = m.start()
+    cut = min(candidates)
     # Walk back to the previous word boundary.
     while cut > 0 and not caption[cut - 1].isspace():
         cut -= 1

{docpluck-2.4.3 → docpluck-2.4.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docpluck"
-version = "2.4.3"
+version = "2.4.4"
 description = "PDF, DOCX, and HTML text extraction and normalization for academic papers"
 readme = "docs/README.md"
 requires-python = ">=3.10"

{docpluck-2.4.3 → docpluck-2.4.4}/tests/test_figure_detect.py

@@ -159,3 +159,62 @@ def test_trim_caption_keeps_minimum_post_label_content():
     out = _trim_caption_at_chart_data(short_pre_label)
     # Sanity check fires; return original.
     assert out == short_pre_label
+
+
+# v2.4.4: caption truncation extended to short-token tick runs (5+ short
+# numeric tokens in a row — axis-tick label sequences from charts).
+
+
+def test_trim_caption_at_tick_run_truncates_axis_labels():
+    """v2.4.4: detect chart axis-tick sequences (5+ short numeric tokens
+    separated only by whitespace) — jama_open_3-style Kaplan-Meier
+    captions absorb gridline values like ``0 0 5 10 15`` that the 6-digit
+    rule didn't catch."""
+    from docpluck.figures.detect import _trim_caption_at_chart_data
+    cap = (
+        "Figure 1. Unadjusted Kaplan-Meier Curves Across Groups With "
+        "Different Objective Sleep Duration for All-Cause Mortality 100 "
+        "90 Survival probability, % 80 70 Sleep duration 60 seven hours "
+        "6 to 7 hours 50 5 to 6 hours less than 5 hours 0 0 5 10 15 "
+        "Follow-up time y No at risk Sleep duration seven hours 340 321 "
+        "280 5 Sleep duration"
+    )
+    out = _trim_caption_at_chart_data(cap)
+    assert "0 0 5 10 15" not in out
+    # Trim should preserve the prose lead-in.
+    assert out.startswith("Figure 1. Unadjusted Kaplan-Meier Curves")
+
+
+def test_trim_caption_preserves_legitimate_prose_with_inline_numbers():
+    """Real caption prose references numbers in stats ('n = 1234', 'p < .001'),
+    but each number is followed by a word — not 5+ stacked numerics in a row."""
+    from docpluck.figures.detect import _trim_caption_at_chart_data
+    cap = (
+        "Figure 2. Mean reaction times across the four experimental "
+        "conditions, with n = 1234 participants total (95% CI [120.5, "
+        "180.3] ms for condition A; 95% CI [110.2, 175.4] for condition "
+        "B). Significant differences observed at p < .001 between paired "
+        "conditions in all 4 contrasts of interest, as predicted."
+    )
+    out = _trim_caption_at_chart_data(cap)
+    assert out == cap
+
+
+def test_trim_caption_picks_earliest_match_across_both_rules():
+    """When both the 6-digit-run and the 5-token-tick rules match,
+    truncate at the earlier offset so we don't keep chart data past the
+    first signal."""
+    from docpluck.figures.detect import _trim_caption_at_chart_data
+    # Tick run appears first; 6-digit run appears later.
+    cap = (
+        "Figure 3. Bar plot of conditions A through F across the years "
+        "of interest 2020 2021 2022 2023 2024 2025 with later analytic "
+        "subsample participant total 4876956 in the secondary cohort "
+        "described in the methods section above and detailed in the "
+        "supplementary materials accompanying this paper."
+    )
+    out = _trim_caption_at_chart_data(cap)
+    # The tick run "2020 2021 2022 2023 2024 2025" appears earlier; trim
+    # there.
+    assert "2020 2021" not in out
+    assert "4876956" not in out