docpluck 2.4.98__tar.gz → 2.4.99__tar.gz

{docpluck-2.4.98 → docpluck-2.4.99}/.claude/skills/docpluck-qa/references/benchmark-mode.md

@@ -12,23 +12,23 @@ Opt-in cross-format benchmark suite (DOCX corpus, DOCX↔PDF parity via Word COM
 - Microsoft Word installed (for `docx2pdf`, accessed via COM)
 - Python packages: `mammoth`, `beautifulsoup4`, `lxml`, `rapidfuzz`, `pdf2docx`, `docx2pdf`
 - CitationGuard corpus present at `C:\Users\filin\Dropbox\Vibe\CitationGuard\apps\worker\testpdfs\validation\docx\`
-- PDF test corpus at `C:\Users\filin\Dropbox\Vibe\PDFextractor\test-pdfs\`
+- PDF test corpus at `C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\test-pdfs\`
 
 ### Running
 
 Full benchmark (5–15 minutes, launches Word):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python benchmarks/run_all.py
 ```
 
 Quick mode (3 files per benchmark, 2–4 minutes):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py --quick
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python benchmarks/run_all.py --quick
 ```
 
 Skip Word-based conversion (if Word unavailable):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py --skip docx2pdf
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python benchmarks/run_all.py --skip docx2pdf
 ```
 
 Individual benchmarks:

{docpluck-2.4.98 → docpluck-2.4.99}/.claude/skills/docpluck-qa/references/check-6-escicheck-local-webapp.md

@@ -7,7 +7,7 @@ Requires local service running on port 6117 AND frontend on port 6116. Test via
 
 First start the service (if not running):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\PDFextractor\service && uvicorn app.main:app --port 6117 --reload &
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\service && uvicorn app.main:app --port 6117 --reload &
 ```
 
 Then run:

{docpluck-2.4.98 → docpluck-2.4.99}/.claude/skills/docpluck-qa/references/check-7-batch-smoke.md

@@ -3,7 +3,7 @@
 _Extracted from [../SKILL.md](../SKILL.md). Full procedure lives here._
 
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\PDFextractor\service && python -c "
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\service && python -c "
 import os
 from docpluck import extract_pdf
 base = '../test-pdfs'

{docpluck-2.4.98 → docpluck-2.4.99}/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## [2.4.99] — 2026-06-29
+
+**Region-driven table capture (DP-1/DP-2): drive Camelot with docpluck's own caption-anchored region as `table_areas` — AI-gold-verified, the first CAPTURE-PATH change in the v2.4.x table series.** `TABLE_EXTRACTION_VERSION` → `2.4.5`. Instead of running Camelot blind (`pages="all"`) and pairing tables to captions after the fact, each table caption now drives a `stream` extraction constrained to its OWN region (`extract_tables_camelot_by_region` + `_region_driven_capture` in `extract_structured.py`), so a caption gets exactly its table by construction. This recovers the **stacked / side-by-side multi-table-per-page** class that blind detection + token-overlap pairing could not separate — **efendic Tables 4+5 now split correctly** (was a 30×2 merge + empty stub), and **12 papers' previously-empty `0×0` stubs become real tables** (chandrashekar, jdm, ieee×4, bmc_med_3 T4, bmj_open_1, …).
+
+Region-driven tables are **candidates, not automatic winners**: `_pick_better_table` keeps the structurally-richer of {region, auto-detect} per caption (more columns + a cell-retention guard), because `_region_for_caption`'s bbox is reliable vertically but often too *narrow* horizontally — without this a 7-column demographics table would collapse to 2 columns (jama_open_1 correctly stays 43×7). A *starved* caption (the case region-driving exists for) has no auto-detect candidate, so the region table wins automatically.
+
+**Content-plausibility guards** (region path only — applying them to auto-detect wholesale rejected legitimate-but-imperfect tables, a regression): `_cell_is_prose` dominance reject (a caption-anchored region that over-captured 2-column body text → cog_emo Table 3 was a 27×4 grid of the procedure paragraph) and `_CAPTION_LABEL_RE` reject (a cell containing `Table N.`/`Figure N.` proves the region absorbed an adjacent caption → cog_emo Table 9). Both now emit honest `0×0` stubs. The caption **page-fix** (advance `char_start` past a leading `\n\n`/`\f` so captions after a paragraph break / at a page top get a non-empty `line_text` and correct page) is the foundation the layout-channel region lookup needs.
+
+Verification (ground truth = AI multimodal `reading`/`stats` golds via article-finder, **never** pdftotext): full **101-PDF structured diff** vs v2.4.98 baseline = 27 papers changed, **0 errors**; **6-paper AI-gold canary** (chandrashekar / ieee_access_4 / efendic page-8 split / bmj_open_1 / ieee_access_7 / cog_emo) all returned FIX / PASS / NET-BETTER with no new regressions. Full table unit + real-PDF test suite green.
+
+**Known documented hard cases** (present at baseline, NOT regressions — a robust fix needs more than this cycle): cog_emo Table 8 (Camelot under-segments two tables into one — the correct content exists in the region candidate but cannot be reliably selected); chandrashekar Table 3/4 caption (2-column caption interleaving in the text channel); efendic Table 2 (Camelot detects zero tables on the page — vector/image grid); efendic Table 1 (tiny glued predictions table, header mis-mark); bmc_med_3 duplicate-fragment pairing (both a same-page raw-text dedupe and a bbox-proximity pairing were tried and **reverted** — each regressed another paper; Camelot bboxes are too unreliable to pair on and dropping a duplicate perturbs the order-dependent pairing — see `LEARNINGS`).
+
 ## [2.4.98] — 2026-06-25
 
 **ESCIcheck handoff defects (DOCPLUCK_HANDOFF_2026-06-25): η²p effect-typing, inline correlation r-typing, and bracketed-CI continuation merge — flatten/cell-cleaning only, AI-gold-verified.** `TABLE_EXTRACTION_VERSION` → `2.4.4`. Render-visible in the inline flattened-table blocks + `.tables.jsonl` sidecar `fields`. **No capture-path change** — caption→table pairing is byte-identical to v2.4.3. Grounded against the AI `reading`/`stats` golds (article-finder), not pdftotext, and confirmed by a 6-paper AI-gold canary verify. Triage: `docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md`.

{docpluck-2.4.98 → docpluck-2.4.99}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.98
+Version: 2.4.99
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://docpluck.app
 Project-URL: Documentation, https://docpluck.app/api-docs

{docpluck-2.4.98 → docpluck-2.4.99}/docpluck/__init__.py

@@ -78,7 +78,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.98"
+__version__ = "2.4.99"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.98 → docpluck-2.4.99}/docpluck/extract_structured.py

@@ -37,7 +37,7 @@ from .tables.render import cells_to_html
 from .telemetry import record_fallback
 
 
-TABLE_EXTRACTION_VERSION = "2.4.4"  # v2.4.4 (ESCIcheck handoff 2026-06-25 — flatten-only, AI-gold-verified): (DP-3) flatten._infer_anova_eta2_hint types a font-dropped η²p effect column by STRUCTURE — an unlabeled estimate column in an F-test/ANOVA results table (F + BF01/CI, no competing d/r/OR) is keyed `eta2`, range-guarded to [0,1]; the η²p glyph itself is unrecoverable (NotoSerif uni:no, OCR-tier). (DP-5b) flatten._inline_stat_field types a self-labeled cell (`r = .67`, `d = 0.32`) by its token even under a generic "Effect size" header. (DP-5a) cell_cleaning._is_fragment_cell recognizes a bracketed-CI close tail (`0.73]`) so a CI split across rows rejoins (`[0.59, 0.73]`) and the junk fragment row is dropped (cog_emo T8 14→10 rows). These three are flatten/cell-cleaning ONLY — no capture-path change, so caption→table pairing is byte-identical to v2.4.3. (DP-1/DP-2 capture recovery — the caption page-attribution fix — was prototyped but REVERTED: AI-gold canary verify showed it mis-pairs same-page-caption tables (efendic T4/T5, cog_emo T8/T9) and only half-fixes plos_med; it is queued as its own gated cycle with same-page disambiguation + region-quality gating. See docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md.) # v2.4.3 (RC-T foundation): tables.whitespace gains a CHAR-LEVEL column-detection fallback (char_whitespace_cells) — when pdfplumber's word grouper glues a tight-kerned numeric row into one token so the word-gap detector finds < 2 columns, recover the grid from char x-gaps, voting on column-START edges (right-aligned data columns are left-edge-stable even when the label column is variable-width) and reinserting intra-cell word spacing from geometry. Fires ONLY as a fallback when the word path returns < 2 columns, so currently-correct tables are byte-unchanged (word path restored verbatim). Recovers ip_feldman Table 10's 7 regression rows in isolation. NOT yet wired to replace a degenerate MATCHED Camelot table (that extract_structured change + region prose-trim is the next gated cycle — see LEARNINGS 2026-06-25). # v2.4.2 (RC-T Layer-2): _extract_table_body_text now (a) Note-anchor — a table's "Note:" footnote is its last element, so trim body prose bled past it (chan_feldman T1/T3, efendic_2022 T5); and (b) degenerate-prose guard — suppress a raw_text fallback that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so render emits a clean caption-only table instead of an unstructured-table dump duplicating Results/Discussion prose (chan_feldman T9 was a verbatim ## Discussion duplicate). FP-safe (real cells start with header/label/number/single-letter marker, never a wrapped continuation); full-corpus 101-PDF guard-diff only trims+suppresses (grew=0 changed=0). # v2.4.1 (DP-2/DP-5): (DP-2) blank-header role recovery now types the unlabeled p-value (a bare `.XXX` after the test stat, no comparison op) and df (a bare integer/Welch-decimal between the stat and the d[CI] column) columns it previously skipped — collabra.77859 T3 fields gain p+df (tables.flatten._recover_blank_roles Pass 4.5). (DP-5) parallel-arm tables with a TWO-ROW header no longer drop their first data row, and a CENTERED super-header is aligned to its arm block instead of its visual-center column: (a) cell_cleaning._is_header_like_row counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, N/A) as data via _DATA_VALUE_CELL_RE so a real first data row isn't read as a 3rd header row (collabra.90203 T10 recovered the Identifiable/Explicit-learning correlation); (b) tables.flatten._detect_column_groups re-derives arm boundaries from equal-width blocks of the data region (each must hold one super-label) so a centered super-label folded mid-span no longer swaps arm values (xiao_2021 T4 Original/Replication F) or pushes a stat column into the label region; (c) tables.flatten._classify_column reads a folded super-header cell's role from its sub-part (collabra.90203 T10 CI). Full-corpus cached-table flatten diff: no clean-table regression. # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
+TABLE_EXTRACTION_VERSION = "2.4.5"  # v2.4.5 (region-driven capture, 2026-06-29 — AI-gold-verified, CAPTURE-PATH change): each table caption now drives Camelot stream with its OWN caption-anchored region as table_areas (extract_tables_camelot_by_region + _region_driven_capture), so a caption gets exactly its table by construction — recovers stacked / side-by-side multi-table-per-page cases blind pages="all" + token-overlap pairing could not separate (efendic Table 4+5 split, 12 papers' 0×0 stubs → real). Region tables are CANDIDATES: _pick_better_table keeps the structurally-richer of {region, auto-detect} per caption (region too-narrow bbox can't collapse a wide table; jama_open_1 stays 43×7). Content-plausibility guards on the region path (_cell_is_prose dominance + _CAPTION_LABEL_RE absorption) reject prose-as-grid / caption-absorbed fragments → honest 0×0 stubs (cog_emo Table 3/9). Caption page-fix (char_start past leading \n\n/\f) is the foundation. Full 101-PDF structured diff (27 changed/0 err) + 6-paper AI-gold canary (all FIX/PASS/NET-BETTER, no new regressions). Known documented hard cases (not regressions vs prior): cog_emo T8 Camelot under-segmentation, chandrashekar 2-col caption interleave, efendic T2 Camelot-invisible, bmc_med_3 duplicate-fragment pairing (dedupe + bbox-proximity pairing both tried + reverted — net-harmful, see LEARNINGS). # v2.4.4 (ESCIcheck handoff 2026-06-25 — flatten-only, AI-gold-verified): (DP-3) flatten._infer_anova_eta2_hint types a font-dropped η²p effect column by STRUCTURE — an unlabeled estimate column in an F-test/ANOVA results table (F + BF01/CI, no competing d/r/OR) is keyed `eta2`, range-guarded to [0,1]; the η²p glyph itself is unrecoverable (NotoSerif uni:no, OCR-tier). (DP-5b) flatten._inline_stat_field types a self-labeled cell (`r = .67`, `d = 0.32`) by its token even under a generic "Effect size" header. (DP-5a) cell_cleaning._is_fragment_cell recognizes a bracketed-CI close tail (`0.73]`) so a CI split across rows rejoins (`[0.59, 0.73]`) and the junk fragment row is dropped (cog_emo T8 14→10 rows). These three are flatten/cell-cleaning ONLY — no capture-path change, so caption→table pairing is byte-identical to v2.4.3. (DP-1/DP-2 capture recovery — the caption page-attribution fix — was prototyped but REVERTED: AI-gold canary verify showed it mis-pairs same-page-caption tables (efendic T4/T5, cog_emo T8/T9) and only half-fixes plos_med; it is queued as its own gated cycle with same-page disambiguation + region-quality gating. See docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md.) # v2.4.3 (RC-T foundation): tables.whitespace gains a CHAR-LEVEL column-detection fallback (char_whitespace_cells) — when pdfplumber's word grouper glues a tight-kerned numeric row into one token so the word-gap detector finds < 2 columns, recover the grid from char x-gaps, voting on column-START edges (right-aligned data columns are left-edge-stable even when the label column is variable-width) and reinserting intra-cell word spacing from geometry. Fires ONLY as a fallback when the word path returns < 2 columns, so currently-correct tables are byte-unchanged (word path restored verbatim). Recovers ip_feldman Table 10's 7 regression rows in isolation. NOT yet wired to replace a degenerate MATCHED Camelot table (that extract_structured change + region prose-trim is the next gated cycle — see LEARNINGS 2026-06-25). # v2.4.2 (RC-T Layer-2): _extract_table_body_text now (a) Note-anchor — a table's "Note:" footnote is its last element, so trim body prose bled past it (chan_feldman T1/T3, efendic_2022 T5); and (b) degenerate-prose guard — suppress a raw_text fallback that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so render emits a clean caption-only table instead of an unstructured-table dump duplicating Results/Discussion prose (chan_feldman T9 was a verbatim ## Discussion duplicate). FP-safe (real cells start with header/label/number/single-letter marker, never a wrapped continuation); full-corpus 101-PDF guard-diff only trims+suppresses (grew=0 changed=0). # v2.4.1 (DP-2/DP-5): (DP-2) blank-header role recovery now types the unlabeled p-value (a bare `.XXX` after the test stat, no comparison op) and df (a bare integer/Welch-decimal between the stat and the d[CI] column) columns it previously skipped — collabra.77859 T3 fields gain p+df (tables.flatten._recover_blank_roles Pass 4.5). (DP-5) parallel-arm tables with a TWO-ROW header no longer drop their first data row, and a CENTERED super-header is aligned to its arm block instead of its visual-center column: (a) cell_cleaning._is_header_like_row counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, N/A) as data via _DATA_VALUE_CELL_RE so a real first data row isn't read as a 3rd header row (collabra.90203 T10 recovered the Identifiable/Explicit-learning correlation); (b) tables.flatten._detect_column_groups re-derives arm boundaries from equal-width blocks of the data region (each must hold one super-label) so a centered super-label folded mid-span no longer swaps arm values (xiao_2021 T4 Original/Replication F) or pushes a stat column into the label region; (c) tables.flatten._classify_column reads a folded super-header cell's role from its sub-part (collabra.90203 T10 CI). Full-corpus cached-table flatten diff: no clean-table regression. # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
 
 TableTextMode = Literal["raw", "placeholder"]
 
@@ -180,12 +180,34 @@ def extract_pdf_structured(
     used_caption_ids: set[int] = set()
     table_captions = [c for c in captions if c.kind == "table"]
 
-    # Filter Camelot's output to tables that have a same-page caption.
-    # This anchors detection to caption signal (matching the pre-pdfplumber-removal
-    # behavior of docpluck) and drops false-positive Camelot detections like
-    # bibliographies or address blocks. Tables without captions are rare in APA
-    # corpus and the existing tests are calibrated against caption-anchored counts.
+    # ---- Region-driven capture (candidate generator) ----
+    # For each caption, hand Camelot the caption-anchored region as table_areas so
+    # it extracts exactly that caption's table — sidestepping the header-absorption,
+    # two-tables-merged-into-one, and caption-pairing failures of blind
+    # auto-detection. CRUCIAL: region-driven tables are CANDIDATES, not winners.
+    # ``_region_for_caption``'s bbox is reliable for VERTICAL anchoring but often
+    # too NARROW horizontally for a wide table (its width tracks the caption /
+    # partial content, not the full grid), so a naive "region wins" would collapse
+    # a 7-column demographics table to 2 columns. We therefore pick, per caption,
+    # the BETTER of the region-driven candidate and the auto-detected candidate
+    # (``_pick_better_table``: more columns wins; tie-break on cell count) — so
+    # region-driving can only ADD coverage (0×0 stub → real table) or replace an
+    # auto-detect table it strictly beats, never degrade one.
+    layout_doc = _layout_doc
+    region_tables: dict[int, Table] = {}
+    if not camelot_disabled and table_captions:
+        region_tables, layout_doc = _region_driven_capture(
+            pdf_bytes, rejoined, table_captions, layout_doc,
+            next_boundary_by_id, method_pieces,
+        )
+
+    # Auto-detect candidate per caption (legacy pairing). Filter Camelot's output
+    # to tables that have a same-page caption — this anchors detection to caption
+    # signal (matching the pre-pdfplumber-removal behavior) and drops false
+    # positives like bibliographies. Uncaptioned auto-detect tables are dropped as
+    # before (rare in the APA corpus; existing tests are caption-anchored).
     pages_with_table_caption = {c.page for c in table_captions}
+    auto_by_cap: dict[int, Table] = {}
     for ct in camelot_tables:
         if (ct.get("page") or 0) not in pages_with_table_caption:
             continue
@@ -196,7 +218,37 @@ def extract_pdf_structured(
             ct["caption"] = _extract_caption_text(
                 rejoined, match, next_boundary_by_id.get(id(match))
             )
-            tables.append(ct)
+            auto_by_cap[id(match)] = ct
+
+    # Pick the candidate per caption. ``used_caption_ids`` already holds the
+    # auto-detect-matched caps; add any caption the region pass alone satisfied.
+    #
+    # ``_pick_better_table`` is the single arbiter: region-driving's value is that
+    # it gives a STARVED caption its own table — and a starved caption has NO
+    # auto-detect candidate (auto-detect captured the page as one table and
+    # token-paired it to the OTHER caption), so ``auto_td is None`` and region
+    # wins automatically. Where auto-detect DID give this caption a real grid, the
+    # column-collapse guard keeps the richer one. This handles both stacked-table
+    # separation (cog_emo Table 9: auto=None → region 5×3 vs baseline 0×0) and
+    # wide single tables (jama_open_1 Table 1: region too narrow → keep auto 43×7)
+    # without a page-multiplicity special case (which over-forced region on
+    # multi-caption pages whose tables auto-detect already separated cleanly —
+    # bmc_med_3, ieee_access_8 — collapsing their columns).
+    region_win = region_only = 0
+    for cap in table_captions:
+        auto_td = auto_by_cap.get(id(cap))
+        region_td = region_tables.get(id(cap))
+        if auto_td is None and region_td is None:
+            continue
+        best = _pick_better_table(region_td, auto_td)
+        if best is region_td and auto_td is not None:
+            region_win += 1
+        elif best is region_td:
+            region_only += 1
+        tables.append(best)
+        used_caption_ids.add(id(cap))
+    if region_win or region_only:
+        method_pieces.append(f"region_pick:{region_win}+{region_only}")
 
     # §A R1 / B1 (NORMALIZATION_VERSION 1.9.23, 2026-05-23): whitespace_cells
     # fallback for caption-detected tables that Camelot could not recover.
@@ -209,8 +261,8 @@ def extract_pdf_structured(
     #
     # Lazy-import the layout extraction + whitespace helper so the rest of
     # the pipeline works in environments without pdfplumber-layout deps; any
-    # failure here falls back transparently to the isolated path.
-    layout_doc = _layout_doc  # reuse caller-supplied doc when available
+    # failure here falls back transparently to the isolated path. `layout_doc`
+    # may already be materialized by the region-driven pass above; reuse it.
     _whitespace_cells = None
     _region_for_caption_fn = None
     unmatched_caps = [
@@ -346,6 +398,176 @@ def _join_split_captions(text: str) -> str:
     return text
 
 
+# A region's bottom is clipped this many points ABOVE the next same-page
+# caption's top, so a short table's region does not extend down into the NEXT
+# table (cog_emo p13: Table 8's matrix region would otherwise absorb Table 9).
+_REGION_NEXT_CAPTION_GAP_PT: float = 2.0
+
+# A region's top is clamped to the caption's top minus this margin (enough to
+# include the caption line's own glyph ascent, but not the content above it), so
+# a region whose SEARCH_ABOVE fallback reached up into a preceding table is
+# pulled back to start at its own caption.
+_REGION_TOP_MARGIN_PT: float = 4.0
+
+
+def _table_cell_count(td: Optional[Table]) -> int:
+    if not td:
+        return 0
+    return sum(1 for c in (td.get("cells") or []) if (c.get("text") or "").strip())
+
+
+def _pick_better_table(
+    region_td: Optional[Table], auto_td: Optional[Table]
+) -> Optional[Table]:
+    """Choose between a caption's region-driven candidate and its auto-detected
+    candidate. Region-driving anchors the RIGHT table to the caption but its
+    bbox can be too narrow (collapsing columns), so it only wins when it is at
+    least as wide as the auto-detect table AND carries comparable content.
+
+    Rules (auto-detect is the incumbent; region must clearly not regress it):
+      - exactly one present → that one.
+      - region has FEWER columns than auto → auto (guards the column-collapse
+        regression: a 7-col demographics table must not become 2-col).
+      - region has MORE columns than auto → region, UNLESS it lost a large share
+        of cells (a wider-but-emptier grid is a mis-detection, keep auto).
+      - equal columns → whichever has more populated cells (ties → auto, the
+        stable incumbent).
+    """
+    if region_td is None:
+        return auto_td
+    if auto_td is None:
+        return region_td
+    rc = int(region_td.get("n_cols") or 0)
+    ac = int(auto_td.get("n_cols") or 0)
+    r_cells = _table_cell_count(region_td)
+    a_cells = _table_cell_count(auto_td)
+    if rc < ac:
+        return auto_td
+    if rc > ac:
+        # Wider grid must still retain most of the content to be believed.
+        if a_cells > 0 and r_cells < 0.6 * a_cells:
+            return auto_td
+        return region_td
+    # Equal column count: prefer the fuller grid; tie goes to the incumbent.
+    return region_td if r_cells > a_cells else auto_td
+
+
+def _region_driven_capture(
+    pdf_bytes: bytes,
+    rejoined: str,
+    table_captions: list[CaptionMatch],
+    layout_doc,
+    next_boundary_by_id: dict[int, Optional[int]],
+    method_pieces: list[str],
+) -> tuple[dict[int, Table], object]:
+    """Drive Camelot once per caption with the caption-anchored region as
+    ``table_areas`` and return ``{id(cap): Table}`` for every caption that
+    yielded a usable table, plus the (possibly newly-materialized) layout doc.
+
+    For each caption we (1) locate its layout region via ``_region_for_caption``,
+    (2) clip the region bottom at the next same-page caption so it can't reach
+    into the following table, (3) convert the pdfplumber TOP-DOWN bbox to the
+    Camelot PDF BOTTOM-UP ``table_areas`` string, and (4) attach the extracted
+    caption text. Any failure degrades gracefully to ``{}`` so the legacy
+    auto-detect + whitespace path still runs.
+    """
+    empty: dict[int, Table] = {}
+    try:
+        from .tables.detect import _region_for_caption, _bbox_of_caption_line
+        from .tables.camelot_extract import extract_tables_camelot_by_region
+        if layout_doc is None:
+            from .extract_layout import extract_pdf_layout
+            layout_doc = extract_pdf_layout(pdf_bytes)
+    except Exception as exc:
+        record_fallback("region_driven_setup_exception", detail=type(exc).__name__)
+        return empty, layout_doc
+
+    pages = getattr(layout_doc, "pages", None) or ()
+
+    # Group captions per page (top-to-bottom) so the next-caption bottom-clip can
+    # look at the following caption on the same page.
+    by_page: dict[int, list[CaptionMatch]] = {}
+    for c in table_captions:
+        by_page.setdefault(c.page, []).append(c)
+
+    specs: list[dict] = []
+    cap_by_key: dict[str, CaptionMatch] = {}
+    for page, caps in by_page.items():
+        if not (1 <= page <= len(pages)):
+            continue
+        page_obj = pages[page - 1]
+        page_height = float(getattr(page_obj, "height", 0.0) or 0.0)
+        if page_height <= 0.0:
+            continue
+        caps_sorted = sorted(caps, key=lambda c: c.char_start)
+        cap_top_td: dict[int, Optional[float]] = {}
+        for c in caps_sorted:
+            cb = _bbox_of_caption_line(page_obj, c)
+            cap_top_td[id(c)] = cb[1] if cb else None
+        for i, c in enumerate(caps_sorted):
+            try:
+                region = _region_for_caption(layout_doc, c)
+            except Exception as exc:
+                record_fallback("region_driven_region_exception", detail=type(exc).__name__)
+                region = None
+            if region is None:
+                continue
+            x0, top, x1, bottom = region.bbox
+            # Clip the region TOP at the caption's own top. A table never extends
+            # ABOVE its caption line; ``_region_for_caption``'s SEARCH_ABOVE
+            # fallback can pull the region up into a PRECEDING table when no
+            # geometry is found below (cog_emo Table 9, whose caption sits below
+            # Table 8's matrix, otherwise captures Table 8's rows). Clamp so the
+            # region starts at the caption line (minus a small margin for the
+            # caption glyphs' own ascent), never higher.
+            this_cap_top = cap_top_td.get(id(c))
+            if this_cap_top is not None and top < this_cap_top - _REGION_TOP_MARGIN_PT:
+                top = this_cap_top - _REGION_TOP_MARGIN_PT
+            # Clip bottom at the next same-page caption that sits below this one.
+            for j in range(i + 1, len(caps_sorted)):
+                nxt_top = cap_top_td.get(id(caps_sorted[j]))
+                if nxt_top is not None and nxt_top > top + 5.0:
+                    bottom = min(bottom, nxt_top - _REGION_NEXT_CAPTION_GAP_PT)
+                    break
+            if bottom <= top:
+                continue
+            key = f"{id(c)}"
+            specs.append({
+                "key": key,
+                "page": page,
+                "area": f"{x0:.2f},{page_height - top:.2f},{x1:.2f},{page_height - bottom:.2f}",
+                "area_bu": (x0, page_height - top, x1, page_height - bottom),
+                "label": c.label,
+            })
+            cap_by_key[key] = c
+
+    if not specs:
+        return empty, layout_doc
+
+    try:
+        by_key = extract_tables_camelot_by_region(pdf_bytes, specs)
+    except Exception as exc:
+        record_fallback("region_driven_extract_exception", detail=type(exc).__name__)
+        return empty, layout_doc
+
+    result: dict[int, Table] = {}
+    for key, td in by_key.items():
+        cap = cap_by_key.get(key)
+        if cap is None:
+            continue
+        # Use the caption-number id scheme (``t{number}``) — the same the
+        # isolated/whitespace path uses — so ids stay deterministic and match the
+        # recognized-prefix invariant (test_table_ids_unique_and_sequential).
+        td["id"] = f"t{cap.number}"
+        td["caption"] = _extract_caption_text(
+            rejoined, cap, next_boundary_by_id.get(id(cap))
+        )
+        result[id(cap)] = td
+    if result:
+        method_pieces.append("region_driven")
+    return result, layout_doc
+
+
 def _find_caption_for_table(
     camelot_table: Table,
     captions: list[CaptionMatch],