docpluck 2.4.97__tar.gz → 2.4.99__tar.gz

{docpluck-2.4.97 → docpluck-2.4.99}/.claude/skills/_project/lessons.md

@@ -641,3 +641,15 @@ Rotation picks `pool[(N mod L) : (N mod L) + rotation_size]` wrapping. Over `cei
 **What (fix):** `api/cron/daily-digest` intermittently logged "Daily dispatch failed" (one-error admin digest email). Ground truth came from prod `system_logs.context.errorStack` (the logger persists `errorName`/`errorMessage`/`errorStack`; queried read-only via the Neon `DATABASE_URL` in `PDFextractor/frontend/.env.local`): two failed runs (06-17, 06-18) were transient **Neon serverless** errors — `terminating connection due to administrator command` (connection recycled on scale-to-zero) and a control-plane HTTP 500 tagged `"neon:retryable":true` — thrown at DIFFERENT queries, with adjacent days succeeding. NOT a logic bug: both unattended crons fire ~15 sequential neon-http calls at 03:00/09:00 UTC against likely-cold compute and the driver does no retries, so one blip aborts the whole run (and Vercel Cron does not re-run failed crons). Fix: new `frontend/src/lib/db-retry.ts` (`withDbRetry`/`isRetryableDbError` — 3 attempts, 250/500ms backoff, retries ONLY transient/connection-class signatures, never app errors) wrapping every DB call in `notifications/dispatch.ts` + `admin/blob-cleanup/route.ts` + the in-band `sendEmail` (safe to replay — `email_events` ON CONFLICT de-dupes). Regression test `frontend/src/lib/db-retry.test.ts` via `node --test` (Node 26 strips TS natively → zero new deps; first frontend unit test + `npm test` script; excluded from `next build` via tsconfig). Shipped commit `ea86e85`; post-deploy `?dryRun=1` smoke = HTTP 200 + clean DispatchResult.
 
 **How to detect next time:** (1) The admin digest aggregates the PREVIOUS 24h, so a "1 error" email naming `api/cron/daily-digest` is usually the prior day's transient blip, not a live outage — before theorizing, query `system_logs` for the real `errorName`/`errorStack` (it's `NeonDbError`, often `neon:retryable:true`). (2) ANY unattended Vercel cron doing a burst of neon-http queries needs `withDbRetry`: Neon scale-to-zero WILL recycle a connection mid-run and there is no driver-level retry. (3) `next build` here does NOT run eslint (Next 16 — see card `nextjs-16-build-does-not-run-eslint`), so a red `npm run lint` does not block deploy; keep the lint gate green separately for `/ship`.
+
+## 2026-06-25 — Branch can lag its own pushed tags; `check_app_pin_sync.py` PASS does NOT prove the branch is current
+
+**Symptom.** Session opened "clean", `check_app_pin_sync.py` PASS — yet `main` (local AND `origin/main`) was at the v2.4.95 commit while tags `v2.4.96`/`v2.4.97` were already pushed 4 commits ahead, the app pin had auto-bumped to `@v2.4.97`, and the working tree `__version__` read 2.4.95. So **production ran v2.4.97 while the library mainline + working tree claimed v2.4.95.** A prior session tagged+pushed without fast-forwarding the branch, and the tree was checked out backwards.
+
+**Why the gate didn't catch it.** `check_app_pin_sync.py` compares the app pin to the *latest tag* — both were v2.4.97, so PASS. It does not compare the *branch HEAD* to the latest tag. And its working-tree note had a direction bug: it told the BEHIND tree it was "ahead — tag+push v2.4.95" (would have tagged an OLDER version). Fixed the note (ordered `_vtuple()`, commit `59cff5b`); the branch-vs-tag check is still a manual step.
+
+**Rule.** At session start (and before any release), run `git tag --sort=-v:refname | head` and `git rev-parse main origin/main` and compare to the latest `v*` tag. If `main` is an ANCESTOR of the latest tag → the branch lags its own release; fast-forward (`git merge --ff-only v<latest>`) before iterating, or you'll work on a stale base and risk re-doing / colliding with shipped work. A green `check_app_pin_sync.py` is necessary but NOT sufficient evidence the branch is current. Companion: card `release-version-collision-with-parallel-uncommitted-stream`.
+
+## 2026-06-25 — Word-multiset preservation is BLIND to reading-order regressions; AI-verify is mandatory before any reorder flip
+
+The RC-1 banded column-correction flag passed a full-corpus word-preservation scan (26/26 baseline papers, 0 multiset violations) — looked safe to flip default-ON. The 8-canary AI-verify then found **3 ON_REGRESSION** the scan was structurally blind to: running-header furniture injected into prose, Abstract/Intro section order inverted, prose fragmented on a *single-column* paper (false-positive gutter). A pure reorder/furniture-injection preserves the word multiset exactly. **Never flip a reading-order-affecting default on word-preservation (or char-ratio / Jaccard) evidence alone — those gates cannot see "right words, wrong order / wrong place." AI-verify against the gold is required, and the bar is zero ON_REGRESSION across the full canary set.** Re-validates the project ground-truth hard rule.

{docpluck-2.4.97 → docpluck-2.4.99}/.claude/skills/docpluck-iterate/LEARNINGS.md

@@ -1327,3 +1327,44 @@ aren't skipped.
 ### Verification Gaps / Deferred (queued, not dropped)
 - **RC-T Layer-1 recovery** — actually RECOVERING lost table data via tight `table_areas` (plos_med T5's 13 SAE rows; chan_feldman T2 column-squish) — out of Option-A scope, deferred by the user.
 - **Audit of the ~37 `_strip_phantom` th-stripped tables** — some title-shaped `<th>` strips may be wrongly-stripped REAL tables (pre-existing, predates this cycle). Needs its own verification cycle (render each, judge real-vs-phantom). Surfaced explicitly; NOT silently shipped around.
+
+---
+
+## Run: 2026-06-25 · cycle 1 (investigation + verification) · @ v2.4.97 · verdict PARTIAL
+
+**Goal:** user invoked `/docpluck-iterate`; established stop-condition implicitly via the two scope decisions below. Targeted RC-T table-bbox (then, on evidence, the RC-1 banded flip).
+
+### What landed (clean, shipped, verified)
+1. **Branch reconciliation (the big one).** At session start `main` (local AND `origin/main`) was at the v2.4.95-era commit `2dbdd98`, but tags `v2.4.96`/`v2.4.97` were already pushed 4 commits AHEAD (real, test-backed RC-T + DP-2/DP-5 work) and the app pin had auto-bumped to `@v2.4.97` → **production ran v2.4.97 while the library mainline claimed v2.4.95.** Fast-forwarded `main` (local + origin) to `813aa4c` (v2.4.97); now mainline == latest tag == app pin. **Lesson: at session start, ALWAYS check `git tag` vs `git rev-parse main origin/main` — a prior session can push tags without advancing the branch, and the working tree can be checked out backwards. The `check_app_pin_sync.py` PASS hid it (pin==tag is true even when the *branch* lags the tag).**
+2. **`check_app_pin_sync.py` direction bug + test** (commit `59cff5b`). `compare()` said "working-tree X is ahead of tag Y — tag+push X" for ANY `X!=Y`; a tree BEHIND its tag (2.4.95 < 2.4.97, exactly this session) got told to tag an OLDER version. Fixed with ordered `_vtuple()`; `tests/test_check_app_pin_sync.py` (10 cases, fails-before/passes-after).
+
+### What was investigated and correctly NOT shipped
+3. **RC-T table-data recovery = the RC-1 banded layout-channel work (architectural).** ip_feldman T10 (and 3 other canary orphans) lose real table data; gold confirms a 7-row table, render drops all 7. Probed FOUR recovery angles (Camelot free-form, whitespace_cells full region, banded text-channel flag, band-clipped whitespace_cells) — each bottoms out in: full-page Camelot bbox / region-overshoot-into-prose / RC-1 interleave INSIDE the table region (table rows + Discussion prose share y-positions) / tight-kerned glued stat cells that defeat x-gap column detection (`pdfplumber_extract_words_unreliable`). The orphan `### Table N` heading is the DELIBERATE v2.4.55 clean-fail (keeps caption for `table_parity`), NOT the bug. **Lesson: reproduce + probe before trusting a "small fix" hypothesis — my B2-is-easy guess was disproven in ~15 min of probing, saving a mis-scoped cycle.** (card `reproduce-triage-defect-at-head`)
+4. **RC-1 banded default-flip ATTEMPTED, REJECTED by 8-canary AI-verify.** Word-preservation corpus scan = 26/26 PRESERVED, 0 violations (looked safe!). But the 8-canary AI-verify (8 parallel Sonnet, Claude Max) found **3 ON_REGRESSION** (deterministically confirmed): `ar_apa` flag-ON injects `M. Muraven / Journal` running-header furniture ×2 + inverts Abstract/Intro order; `chan_feldman` injects Power-Analysis prose into Extension; `maier` (SINGLE-COLUMN) fragments prose via a false-positive gutter. **THE LESSON (re-validates the project hard rule): the word-multiset gate is BLIND to reordering + furniture-injection — it passed 26/26 while the flag corrupted 3 canaries. AI-verify is necessary, not optional, before any reading-order default flip.** Flag stays default-OFF. Precise gap-5/6/7 blockers + a stricter zero-regression re-verify gate written to the RC-1 spec.
+
+### Verification-infra defect re-confirmed (surfaced, not fixed — needs portfolio decision)
+5. **The pre-push canary-audit.sh hook clobbered run-meta `phase_5d_runs` + `cycle_status` on BOTH pushes this session** — overwriting my honest FAIL verdicts with `AUDIT_DEFERRED_TO_AGENT → union PASS` placeholders (`cycle_status` flipped to PASS, a broken corpus reading green). Restored truthful verdicts after each push per memory `feedback_canary_audit_clobbers_phase5d`. **This bug actively manufactures false-green on a FAIL corpus; it bit twice in one session.** The fix (make the hook not write deferred-PASS, or write BLOCKED/AUDIT_DEFERRED instead of PASS) is a substrate change in `~/.claude/skills/_shared/iterate-loop/canary-audit.sh` — surfaced for a dedicated substrate cycle.
+
+### Open (queued, NOT dropped — rule 0e-bis)
+- 4 canary FAILs remain (ip_feldman T10, chan_feldman T6/T9, chandrashekar T2 table-data loss; plos_med sidebar interleave) → the RC-1 banded layout-channel + sidebar-interleave architectural cycle (spec `2026-06-08-rc1-region-aware-column-architecture.md`, sharpened this run).
+- Run verdict: **PARTIAL** — 2 clean fixes shipped; the canary corpus is NOT clean (honest punch-list above), so the standing corpus verdict is FAIL and the iterate-gate `--close` correctly cannot green-close.
+
+---
+
+## Run: 2026-06-25 (cont.) · RC-T char-level column recovery — FOUNDATION landed (v2.4.98), T10 wiring deferred
+
+### Outcome
+- **char_whitespace_cells (the char-level absolute-x-gap column detector) IMPLEMENTED + tested** in `docpluck/tables/whitespace.py`, wired as an automatic fallback inside `whitespace_cells` when the word path finds < 2 columns. This is the layout-channel piece that finally makes the "RC-T table-data recovery" tractable.
+
+### Blind spot closed (the key technical finding)
+- **Earlier this session I assessed T10 recovery as "architectural / multi-session, char detection too hard."** That was WRONG — disproven by probing char-level x-positions: ip_feldman Table 10's 3 stat columns are cleanly separable at the CHAR level (gaps 23-78pt) even though pdfplumber's WORD grouper glues the whole numeric row (`.29***−.21***.07`) into one token so the word-gap detector finds nothing. This is exactly the `pdfplumber_extract_words_unreliable` lesson ("always carry a char-level absolute-x-gap fallback") — I should have reached for it before declaring the wall.
+- **Right-aligned numeric tables need column-START-edge voting, not gap-MIDPOINT voting.** The label column is variable-width (`Loneliness` vs `Social orientation scale`), so gap midpoints scatter and never reach the stability threshold; but the DATA columns are left-aligned to fixed x, so their left edges are stable. `_find_stable_column_boundaries(bucket_pt>0)` votes on column starts + buckets within 8pt. With this, char_whitespace_cells recovers all 7 T10 data rows matching the gold exactly.
+
+### Edge case
+- **The word path MUST stay byte-identical when adding a char path.** First cut refactored `_find_stable_column_boundaries` to count only multi-column rows in the denominator — that silently changed the WORD path's stability threshold and broke 2 existing table tests (caught at ~65% of the broad run). Fix: branch the function — `bucket_pt<=0` runs the verbatim original word logic, `bucket_pt>0` runs the new char logic. Lesson: when extending a shared helper, keep the existing caller's path provably unchanged (literal copy), don't "improve" it in the same edit.
+
+### Deferred (queued, NOT dropped — rule 0e-bis)
+- **char_whitespace_cells alone does NOT close the T10 canary in production.** T10's caption is *matched* by the garbage full-page Camelot table, so it's added directly to `tables` and the whitespace fallback (where the char path lives) is never reached for it. Closing T10 needs the **degenerate-Camelot-table replacement** in `extract_structured.py`: when a Camelot-matched table has a full-page-ish bbox + furniture/prose cells, discard it and re-extract via `whitespace_cells` on the gutter-band-clipped caption region, PLUS a region prose-trim (the `(extension)` wrapped-caption + Discussion-prose tail still contaminate the region). That change touches the core extraction path for ALL papers → requires the full ~48-paper guard-live-vs-bypassed diff + 7-canary AI-verify before it ships (cycle-3 caption-follows revert is the precedent). Foundation is in place; this is the next focused cycle.
+
+### Verification
+- `tests/test_whitespace_char_fallback.py` (5 cases incl. real-PDF ip_feldman T10 = all 7 gold rows recovered; synthetic tight-kerned recovery; word-space reinsertion; delegation wiring; single-column-no-fabrication guard). 15/15 pass on the whitespace/char/caption-table suite; 0 failures across the broad table/render run; word path proven byte-identical (restored verbatim).

{docpluck-2.4.97 → docpluck-2.4.99}/.claude/skills/docpluck-qa/references/benchmark-mode.md

@@ -12,23 +12,23 @@ Opt-in cross-format benchmark suite (DOCX corpus, DOCX↔PDF parity via Word COM
 - Microsoft Word installed (for `docx2pdf`, accessed via COM)
 - Python packages: `mammoth`, `beautifulsoup4`, `lxml`, `rapidfuzz`, `pdf2docx`, `docx2pdf`
 - CitationGuard corpus present at `C:\Users\filin\Dropbox\Vibe\CitationGuard\apps\worker\testpdfs\validation\docx\`
-- PDF test corpus at `C:\Users\filin\Dropbox\Vibe\PDFextractor\test-pdfs\`
+- PDF test corpus at `C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\test-pdfs\`
 
 ### Running
 
 Full benchmark (5–15 minutes, launches Word):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python benchmarks/run_all.py
 ```
 
 Quick mode (3 files per benchmark, 2–4 minutes):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py --quick
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python benchmarks/run_all.py --quick
 ```
 
 Skip Word-based conversion (if Word unavailable):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\docpluck && python benchmarks/run_all.py --skip docx2pdf
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck && python benchmarks/run_all.py --skip docx2pdf
 ```
 
 Individual benchmarks:

{docpluck-2.4.97 → docpluck-2.4.99}/.claude/skills/docpluck-qa/references/check-6-escicheck-local-webapp.md

@@ -7,7 +7,7 @@ Requires local service running on port 6117 AND frontend on port 6116. Test via
 
 First start the service (if not running):
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\PDFextractor\service && uvicorn app.main:app --port 6117 --reload &
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\service && uvicorn app.main:app --port 6117 --reload &
 ```
 
 Then run:

{docpluck-2.4.97 → docpluck-2.4.99}/.claude/skills/docpluck-qa/references/check-7-batch-smoke.md

@@ -3,7 +3,7 @@
 _Extracted from [../SKILL.md](../SKILL.md). Full procedure lives here._
 
 ```bash
-cd C:\Users\filin\Dropbox\Vibe\PDFextractor\service && python -c "
+cd C:\Users\filin\Dropbox\Vibe\MetaScienceTools\PDFextractor\service && python -c "
 import os
 from docpluck import extract_pdf
 base = '../test-pdfs'

{docpluck-2.4.97 → docpluck-2.4.99}/CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## [2.4.99] — 2026-06-29
+
+**Region-driven table capture (DP-1/DP-2): drive Camelot with docpluck's own caption-anchored region as `table_areas` — AI-gold-verified, the first CAPTURE-PATH change in the v2.4.x table series.** `TABLE_EXTRACTION_VERSION` → `2.4.5`. Instead of running Camelot blind (`pages="all"`) and pairing tables to captions after the fact, each table caption now drives a `stream` extraction constrained to its OWN region (`extract_tables_camelot_by_region` + `_region_driven_capture` in `extract_structured.py`), so a caption gets exactly its table by construction. This recovers the **stacked / side-by-side multi-table-per-page** class that blind detection + token-overlap pairing could not separate — **efendic Tables 4+5 now split correctly** (was a 30×2 merge + empty stub), and **12 papers' previously-empty `0×0` stubs become real tables** (chandrashekar, jdm, ieee×4, bmc_med_3 T4, bmj_open_1, …).
+
+Region-driven tables are **candidates, not automatic winners**: `_pick_better_table` keeps the structurally-richer of {region, auto-detect} per caption (more columns + a cell-retention guard), because `_region_for_caption`'s bbox is reliable vertically but often too *narrow* horizontally — without this a 7-column demographics table would collapse to 2 columns (jama_open_1 correctly stays 43×7). A *starved* caption (the case region-driving exists for) has no auto-detect candidate, so the region table wins automatically.
+
+**Content-plausibility guards** (region path only — applying them to auto-detect wholesale rejected legitimate-but-imperfect tables, a regression): `_cell_is_prose` dominance reject (a caption-anchored region that over-captured 2-column body text → cog_emo Table 3 was a 27×4 grid of the procedure paragraph) and `_CAPTION_LABEL_RE` reject (a cell containing `Table N.`/`Figure N.` proves the region absorbed an adjacent caption → cog_emo Table 9). Both now emit honest `0×0` stubs. The caption **page-fix** (advance `char_start` past a leading `\n\n`/`\f` so captions after a paragraph break / at a page top get a non-empty `line_text` and correct page) is the foundation the layout-channel region lookup needs.
+
+Verification (ground truth = AI multimodal `reading`/`stats` golds via article-finder, **never** pdftotext): full **101-PDF structured diff** vs v2.4.98 baseline = 27 papers changed, **0 errors**; **6-paper AI-gold canary** (chandrashekar / ieee_access_4 / efendic page-8 split / bmj_open_1 / ieee_access_7 / cog_emo) all returned FIX / PASS / NET-BETTER with no new regressions. Full table unit + real-PDF test suite green.
+
+**Known documented hard cases** (present at baseline, NOT regressions — a robust fix needs more than this cycle): cog_emo Table 8 (Camelot under-segments two tables into one — the correct content exists in the region candidate but cannot be reliably selected); chandrashekar Table 3/4 caption (2-column caption interleaving in the text channel); efendic Table 2 (Camelot detects zero tables on the page — vector/image grid); efendic Table 1 (tiny glued predictions table, header mis-mark); bmc_med_3 duplicate-fragment pairing (both a same-page raw-text dedupe and a bbox-proximity pairing were tried and **reverted** — each regressed another paper; Camelot bboxes are too unreliable to pair on and dropping a duplicate perturbs the order-dependent pairing — see `LEARNINGS`).
+
+## [2.4.98] — 2026-06-25
+
+**ESCIcheck handoff defects (DOCPLUCK_HANDOFF_2026-06-25): η²p effect-typing, inline correlation r-typing, and bracketed-CI continuation merge — flatten/cell-cleaning only, AI-gold-verified.** `TABLE_EXTRACTION_VERSION` → `2.4.4`. Render-visible in the inline flattened-table blocks + `.tables.jsonl` sidecar `fields`. **No capture-path change** — caption→table pairing is byte-identical to v2.4.3. Grounded against the AI `reading`/`stats` golds (article-finder), not pdftotext, and confirmed by a 6-paper AI-gold canary verify. Triage: `docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md`.
+
+- **DP-3 (table side) — type a font-dropped partial-η² effect column.** Many APA ANOVA tables report effect size as `η²p`, but the glyph lives in a font with no ToUnicode mapping (`pdffonts`: `uni:no`), so pdftotext AND pdfplumber both decode it as a space — the effect-size column header is blank and no "eta"/"η" token survives anywhere, so `tables.flatten._effect_key` fell back to the generic `est` key and the consumer got a nameless number. `flatten._infer_anova_eta2_hint` now types an unlabeled estimate column as `eta2` when the table is structurally an F-test/ANOVA results table (carries an `F` column + a `BF01` and/or CI column) and names NO competing effect (`d`/`dz`/`r`/`OR`/`g`); the value is range-guarded to η²'s domain `[0, 1]`. Keyed on the structural signature, never paper identity. `collabra.90203` Tables 8 & 9 now emit `eta2` matching the gold's `η²p` column (`.06, .00, .01, .04, …`). **The text-channel η²p (in body prose) is unrecoverable — OCR-tier won't-fix** (glyph identity absent from the PDF), as is DP-6 figure-inset text.
+
+- **DP-5 (b) — type inline self-labeled correlation cells.** A cell that states its own statistic — `r = .67`, `r = -.73`, `d = 0.32` — is now typed by that token even when the column header is generic ("Effect size") and the column-role recovery produced no estimate for it (`flatten._inline_stat_field`). `cog_emo` (`10.1080/02699931.2024.2434156`) Table 8's intercorrelation rows now carry a typed `r` with their CI/p, matching the gold. The display value is the number only, so the sentence renders `r = 0.63` (not `r = r = .67`).
+
+- **DP-5 (a) — merge a CI split across two rows by its bracket tail.** Camelot stream splits a wrapped CI cell — `[0.59,` on one row, `0.73]` on the next. `cell_cleaning._merge_continuation_rows` already folded the *parenthetical* tail (`8.34)`) but not the *bracketed* CI tail (`0.73]`); `_is_fragment_cell` now recognizes a close-bracket tail, so the bound rejoins (`[0.59, 0.73]`) and the bare-fragment junk row no longer survives (cog_emo Table 8: 14 → 10 rows, CIs complete).
+
+**Deferred (prototyped + REVERTED this cycle — surfaced, not dropped): DP-1 / DP-2 capture recovery.** A caption page-attribution fix (`find_caption_matches` advancing `char_start` past the leading `\f` its `^\s*` ate) does correct the off-by-one that mis-pages a table starting a page, and in isolation it unblocked `collabra.77859` Table 4's replication `t/df/d` (DP-1). But the **mandatory AI-gold canary verify showed it is net-harmful corpus-wide**: populating the previously-empty `line_text` re-scores `_find_caption_for_table`'s same-page token overlap and surfaces low-quality whitespace tables, mis-pairing tables whose captions share a page (efendic T4/T5, cog_emo T8/T9 swapped) and only half-fixing plos_med (Table 4 still wrong, Table 5 degraded). Reverted to keep the corpus regression-free. It needs same-page-caption disambiguation + whitespace-region quality gating, queued as its own gated cycle (full 48-paper guard-diff + canary AI-verify). DP-2's Table 1 (an 8-column grouped table) is beyond the lineless fallback regardless. See the triage doc.
+
+**Also fixed (pre-existing, leave-nothing-behind):** `test_table_html_renders_when_structured` and `test_table_kinds_are_valid` asserted a non-structured table must be `kind == "isolated"`, but the layout-channel `whitespace` fallback (a valid kind since the v2.4.3 foundation) can surface — including when Camelot returns "no tables" under cumulative test-suite load (memory `feedback_camelot_flake_cumulative_load`). The tests now accept `whitespace` (asserting cells + HTML present, no Camelot confidence), so a Camelot load-flake on the jama/ieee/chen fixtures no longer mis-fails them.
+
+Verification: shipped flatten/cell-cleaning changes covered by `tests/test_tables_flatten.py` (TestInlineSelfLabeledStat ×3, TestBracketTailCIContinuation) + `tests/test_tables_flatten_blank_header_recovery.py::TestT90203Table8BlankHeaders` (eta2 typed, est gone). Full library suite green under `pytest -n10` (real-PDF Camelot tests flake only under serial cumulative load — each passes per-file / under xdist). 6-paper AI-gold canary verify confirms DP-3 (collabra.90203 η²p), DP-5 (cog_emo r + CI) correct; it is what caught the DP-1/DP-2 page-fix regression and drove its revert.
+
+---
+
+**RC-T foundation (earlier same-day commit): char-level column-detection fallback for tight-kerned tables.** `TABLE_EXTRACTION_VERSION` → `2.4.3` (now → `2.4.4` per above). Library-internal capability addition — a foundation for the still-open RC-T Layer-1 table-data recovery.
+
+- **`docpluck/tables/whitespace.py` gains `char_whitespace_cells`** — a char-level absolute-x-gap column detector, wired as an automatic fallback inside `whitespace_cells` that fires ONLY when the word-gap detector finds < 2 columns. On tight-kerned PDFs pdfplumber's word grouper glues a whole numeric row into one token (e.g. ip_feldman Table 10's `.29***−.21***.07`), so the word path finds no column gaps; the chars are still cleanly separated by large inter-column gaps. The fallback recovers the grid from char x-gaps, **voting on column-START edges** (in a right-aligned numeric table the label column is variable-width so gap midpoints scatter, but the data columns are left-aligned to fixed x), with an 8pt boundary bucket, and reinserts intra-cell word spacing from glyph geometry.
+- **Strictly additive — currently-correct tables are byte-unchanged.** The word path (`_find_stable_column_boundaries(bucket_pt=0)`) is restored verbatim to its pre-change behavior; only a region that previously yielded NO whitespace grid (`whitespace_cells` returned `[]`) can now gain one, and only when char-columns clear a 60%-row stability bar. In isolation this recovers all 7 of ip_feldman Table 10's regression-coefficient rows matching the AI gold.
+
+**Not yet closed (next gated cycle):** the char fallback alone does NOT fix Table 10 in the live pipeline — T10's caption is *matched* by a degenerate full-page Camelot table, which is used directly, so the whitespace fallback (where the char path lives) is never reached. Closing it needs an `extract_structured.py` change — detect a degenerate matched Camelot table (full-page-ish bbox + furniture/prose cells), discard it, and re-extract via `whitespace_cells` on the gutter-band-clipped caption region, plus a region prose-trim — which touches the core extraction path for all papers and so requires the full ~48-paper guard-diff + 7-canary AI-verify before it ships. See `LEARNINGS.md` 2026-06-25 and `docs/superpowers/specs/2026-06-21-rc-t-table-region-prose-contamination.md`.
+
+Verification: `tests/test_whitespace_char_fallback.py` (5 cases incl. real-PDF ip_feldman Table 10 = all 7 gold rows recovered, synthetic tight-kerned recovery, intra-cell word-space reinsertion, delegation wiring, single-column-no-fabrication guard); 15/15 pass on the whitespace/char/caption-table suite across two independent runs; 0 failures across the broad table/render suite; word path proven byte-identical.
+
 ## [2.4.97] — 2026-06-22
 
 **Three table fixes shipped together (combined from two concurrent sessions): type the skipped p+df columns (DP-2), stop dropping / mis-binding two-header-row tables (DP-5), and stop the table raw_text fallback swallowing body prose (RC-T Layer-2).** `TABLE_EXTRACTION_VERSION` → `2.4.2`; no `NORMALIZATION_VERSION` / `SECTIONING_VERSION` change. DP-2/DP-5 are render-visible in the inline flattened-table blocks + the `.tables.jsonl` sidecar `fields` (the `<table>` HTML gains the previously-dropped data rows); RC-T Layer-2 is render-visible in the `unstructured-table` fallback blocks. DP-2/DP-5 filed in `ESCIcheckapp/docs/DOCPLUCK_HANDOFF_2026-06-21.md`; RC-T Layer-2 per `docs/superpowers/specs/2026-06-21-rc-t-table-region-prose-contamination.md`.

{docpluck-2.4.97 → docpluck-2.4.99}/LESSONS.md

@@ -274,6 +274,74 @@ Cite: `docpluck/tables/camelot_extract.py::_augment_lattice_with_stream_rows` +
 
 ---
 
+## L-010 — A caption that starts a page is mis-paged by the `^\s*` form-feed; and a font with no ToUnicode makes its glyph unrecoverable (recover the column ROLE, not the glyph)
+
+### Two findings, both surfaced 2026-06-25 by the ESCIcheck handoff
+
+1. **The `^\s*` caption regex eats the `\f` and mis-pages a page-starting table.**
+   `TABLE_CAPTION_RE` / `FIGURE_CAPTION_RE` begin `^\s*`. When a table starts a new
+   page, pdftotext emits `…results\n\fTable 4. …`, and `\s*` consumes the `\f`, so
+   `m.start()` lands *before* the form-feed. `_page_for_offset` then counts the
+   caption on the page BEFORE the break (off-by-one), and `_line_at` returns the
+   empty pre-`\f` segment (so `line_text == ''`). With the wrong page,
+   `_bbox_of_caption_line` can't find the caption in the layout channel → the whole
+   layout-region lookup returns None → the whitespace/char fallback never fires and
+   the table degrades to a caption-only stub. On `collabra.77859` this hit **all 5
+   tables**. The seemingly-obvious fix — advance `char_start` past the leading `\f`
+   to the actual "Table"/"Figure" token (`captions.find_caption_matches`) — DOES
+   correct the page AND unblock `collabra.77859` Table 4's replication stats (DP-1) in
+   isolation. **It was tried 2026-06-25 and REVERTED.** Populating the
+   previously-empty `line_text` re-scores `_find_caption_for_table`'s same-page
+   token-overlap and surfaces low-quality whitespace tables, so the mandatory AI-gold
+   canary verify caught it mis-pairing tables whose captions share a page (efendic
+   T4/T5, cog_emo T8/T9 swapped) and only half-fixing plos_med. **The lesson is the
+   process, not the patch:** a capture-path change that helps one paper in isolation
+   can silently mis-pair others — only a corpus-wide AI-gold verify (NOT the unit
+   suite, which stayed green) reveals it. The real fix needs same-page-caption
+   disambiguation in `_find_caption_for_table` + whitespace-region quality gating
+   FIRST; queued as its own gated cycle. Symptom to watch: `line_text == ''` on a
+   caption, or a caption whose `page` is one less than where the table visibly is.
+
+2. **A glyph with no ToUnicode mapping is gone from BOTH channels — recover its
+   ROLE, not the glyph.** `collabra.90203` reports `η²p`, but `pdffonts` shows the
+   symbol font as `uni: no` (no ToUnicode CMap), so pdftotext AND pdfplumber both
+   decode the glyph as U+0020 — the text reads `( = .000, …)` and the table's
+   effect-column header is blank. This is the same class as the residual deleted-minus
+   (memory `project_docpluck_rc_b7_done_w0h`): the *character identity* is absent from
+   the PDF, so text-channel recovery is **OCR-tier won't-fix**. But in a TABLE the
+   column's *role* is recoverable from structure — an F-test/ANOVA results table that
+   reports a Bayes factor + CI and names no competing effect reports η²p by APA
+   convention — so type the value `eta2` from the structural signature (range-guarded
+   to η²'s `[0,1]` domain) even though the glyph is unrecoverable
+   (`flatten._infer_anova_eta2_hint`). Don't chase the glyph; recover the meaning.
+
+### The rules
+
+1. **A caption-page / capture-path change MUST be AI-gold-verified across the corpus
+   before shipping — the unit suite will not catch a mis-pairing.** The page-fix kept
+   all 1852 unit tests green yet swapped table↔caption pairings on 3 papers. Per the
+   project ground-truth rule, render the canary set and compare TABLES against the AI
+   `reading` golds; revert if any paper regresses. (`^\s*`-anchored scans that skip
+   `\f` ARE the right idea for page attribution, but the downstream `line_text` /
+   same-page-caption scoring must be made robust in the same change, not after.)
+2. **Before "the symbol got stripped", run `pdffonts`.** If the glyph's font is
+   `uni:no`, nothing in the byte stream carries its identity — stop trying to recover
+   the glyph; recover the column's role/meaning from structure, or mark it OCR-tier.
+   (Shipped: `flatten._infer_anova_eta2_hint` types the value `eta2` from the F-test
+   table structure even though the η²p glyph is gone.)
+3. **A self-labeled cell beats its column header.** `r = .67` / `d = 0.32` states its
+   own type; type by the cell token even under a generic "Effect size" header
+   (`flatten._inline_stat_field`, shipped) — store only the numeric part so the
+   sentence assembler doesn't double the prefix (`r = r = .67`).
+
+Cite (SHIPPED v2.4.4): `docpluck/tables/flatten.py::_infer_anova_eta2_hint` +
+`_inline_stat_field`, `docpluck/tables/cell_cleaning.py::_is_fragment_cell` (bracket-CI
+tail). REVERTED (queued): the `captions.find_caption_matches` char_start advance +
+`whitespace._whitespace_grid_is_clean` / `_trim_trailing_prose_rows` gates.
+See `docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md`, CHANGELOG v2.4.98.
+
+---
+
 ## When to add a new lesson here
 
 Add a lesson when:

{docpluck-2.4.97 → docpluck-2.4.99}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.97
+Version: 2.4.99
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://docpluck.app
 Project-URL: Documentation, https://docpluck.app/api-docs

{docpluck-2.4.97 → docpluck-2.4.99}/docpluck/__init__.py

@@ -78,7 +78,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.97"
+__version__ = "2.4.99"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.97 → docpluck-2.4.99}/docpluck/extract_structured.py

@@ -37,7 +37,7 @@ from .tables.render import cells_to_html
 from .telemetry import record_fallback
 
 
-TABLE_EXTRACTION_VERSION = "2.4.2"  # v2.4.2 (RC-T Layer-2): _extract_table_body_text now (a) Note-anchor — a table's "Note:" footnote is its last element, so trim body prose bled past it (chan_feldman T1/T3, efendic_2022 T5); and (b) degenerate-prose guard — suppress a raw_text fallback that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so render emits a clean caption-only table instead of an unstructured-table dump duplicating Results/Discussion prose (chan_feldman T9 was a verbatim ## Discussion duplicate). FP-safe (real cells start with header/label/number/single-letter marker, never a wrapped continuation); full-corpus 101-PDF guard-diff only trims+suppresses (grew=0 changed=0). # v2.4.1 (DP-2/DP-5): (DP-2) blank-header role recovery now types the unlabeled p-value (a bare `.XXX` after the test stat, no comparison op) and df (a bare integer/Welch-decimal between the stat and the d[CI] column) columns it previously skipped — collabra.77859 T3 fields gain p+df (tables.flatten._recover_blank_roles Pass 4.5). (DP-5) parallel-arm tables with a TWO-ROW header no longer drop their first data row, and a CENTERED super-header is aligned to its arm block instead of its visual-center column: (a) cell_cleaning._is_header_like_row counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, N/A) as data via _DATA_VALUE_CELL_RE so a real first data row isn't read as a 3rd header row (collabra.90203 T10 recovered the Identifiable/Explicit-learning correlation); (b) tables.flatten._detect_column_groups re-derives arm boundaries from equal-width blocks of the data region (each must hold one super-label) so a centered super-label folded mid-span no longer swaps arm values (xiao_2021 T4 Original/Replication F) or pushes a stat column into the label region; (c) tables.flatten._classify_column reads a folded super-header cell's role from its sub-part (collabra.90203 T10 CI). Full-corpus cached-table flatten diff: no clean-table regression. # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
+TABLE_EXTRACTION_VERSION = "2.4.5"  # v2.4.5 (region-driven capture, 2026-06-29 — AI-gold-verified, CAPTURE-PATH change): each table caption now drives Camelot stream with its OWN caption-anchored region as table_areas (extract_tables_camelot_by_region + _region_driven_capture), so a caption gets exactly its table by construction — recovers stacked / side-by-side multi-table-per-page cases blind pages="all" + token-overlap pairing could not separate (efendic Table 4+5 split, 12 papers' 0×0 stubs → real). Region tables are CANDIDATES: _pick_better_table keeps the structurally-richer of {region, auto-detect} per caption (region too-narrow bbox can't collapse a wide table; jama_open_1 stays 43×7). Content-plausibility guards on the region path (_cell_is_prose dominance + _CAPTION_LABEL_RE absorption) reject prose-as-grid / caption-absorbed fragments → honest 0×0 stubs (cog_emo Table 3/9). Caption page-fix (char_start past leading \n\n/\f) is the foundation. Full 101-PDF structured diff (27 changed/0 err) + 6-paper AI-gold canary (all FIX/PASS/NET-BETTER, no new regressions). Known documented hard cases (not regressions vs prior): cog_emo T8 Camelot under-segmentation, chandrashekar 2-col caption interleave, efendic T2 Camelot-invisible, bmc_med_3 duplicate-fragment pairing (dedupe + bbox-proximity pairing both tried + reverted — net-harmful, see LEARNINGS). # v2.4.4 (ESCIcheck handoff 2026-06-25 — flatten-only, AI-gold-verified): (DP-3) flatten._infer_anova_eta2_hint types a font-dropped η²p effect column by STRUCTURE — an unlabeled estimate column in an F-test/ANOVA results table (F + BF01/CI, no competing d/r/OR) is keyed `eta2`, range-guarded to [0,1]; the η²p glyph itself is unrecoverable (NotoSerif uni:no, OCR-tier). (DP-5b) flatten._inline_stat_field types a self-labeled cell (`r = .67`, `d = 0.32`) by its token even under a generic "Effect size" header. (DP-5a) cell_cleaning._is_fragment_cell recognizes a bracketed-CI close tail (`0.73]`) so a CI split across rows rejoins (`[0.59, 0.73]`) and the junk fragment row is dropped (cog_emo T8 14→10 rows). These three are flatten/cell-cleaning ONLY — no capture-path change, so caption→table pairing is byte-identical to v2.4.3. (DP-1/DP-2 capture recovery — the caption page-attribution fix — was prototyped but REVERTED: AI-gold canary verify showed it mis-pairs same-page-caption tables (efendic T4/T5, cog_emo T8/T9) and only half-fixes plos_med; it is queued as its own gated cycle with same-page disambiguation + region-quality gating. See docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md.) # v2.4.3 (RC-T foundation): tables.whitespace gains a CHAR-LEVEL column-detection fallback (char_whitespace_cells) — when pdfplumber's word grouper glues a tight-kerned numeric row into one token so the word-gap detector finds < 2 columns, recover the grid from char x-gaps, voting on column-START edges (right-aligned data columns are left-edge-stable even when the label column is variable-width) and reinserting intra-cell word spacing from geometry. Fires ONLY as a fallback when the word path returns < 2 columns, so currently-correct tables are byte-unchanged (word path restored verbatim). Recovers ip_feldman Table 10's 7 regression rows in isolation. NOT yet wired to replace a degenerate MATCHED Camelot table (that extract_structured change + region prose-trim is the next gated cycle — see LEARNINGS 2026-06-25). # v2.4.2 (RC-T Layer-2): _extract_table_body_text now (a) Note-anchor — a table's "Note:" footnote is its last element, so trim body prose bled past it (chan_feldman T1/T3, efendic_2022 T5); and (b) degenerate-prose guard — suppress a raw_text fallback that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so render emits a clean caption-only table instead of an unstructured-table dump duplicating Results/Discussion prose (chan_feldman T9 was a verbatim ## Discussion duplicate). FP-safe (real cells start with header/label/number/single-letter marker, never a wrapped continuation); full-corpus 101-PDF guard-diff only trims+suppresses (grew=0 changed=0). # v2.4.1 (DP-2/DP-5): (DP-2) blank-header role recovery now types the unlabeled p-value (a bare `.XXX` after the test stat, no comparison op) and df (a bare integer/Welch-decimal between the stat and the d[CI] column) columns it previously skipped — collabra.77859 T3 fields gain p+df (tables.flatten._recover_blank_roles Pass 4.5). (DP-5) parallel-arm tables with a TWO-ROW header no longer drop their first data row, and a CENTERED super-header is aligned to its arm block instead of its visual-center column: (a) cell_cleaning._is_header_like_row counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, N/A) as data via _DATA_VALUE_CELL_RE so a real first data row isn't read as a 3rd header row (collabra.90203 T10 recovered the Identifiable/Explicit-learning correlation); (b) tables.flatten._detect_column_groups re-derives arm boundaries from equal-width blocks of the data region (each must hold one super-label) so a centered super-label folded mid-span no longer swaps arm values (xiao_2021 T4 Original/Replication F) or pushes a stat column into the label region; (c) tables.flatten._classify_column reads a folded super-header cell's role from its sub-part (collabra.90203 T10 CI). Full-corpus cached-table flatten diff: no clean-table regression. # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
 
 TableTextMode = Literal["raw", "placeholder"]
 
@@ -180,12 +180,34 @@ def extract_pdf_structured(
     used_caption_ids: set[int] = set()
     table_captions = [c for c in captions if c.kind == "table"]
 
-    # Filter Camelot's output to tables that have a same-page caption.
-    # This anchors detection to caption signal (matching the pre-pdfplumber-removal
-    # behavior of docpluck) and drops false-positive Camelot detections like
-    # bibliographies or address blocks. Tables without captions are rare in APA
-    # corpus and the existing tests are calibrated against caption-anchored counts.
+    # ---- Region-driven capture (candidate generator) ----
+    # For each caption, hand Camelot the caption-anchored region as table_areas so
+    # it extracts exactly that caption's table — sidestepping the header-absorption,
+    # two-tables-merged-into-one, and caption-pairing failures of blind
+    # auto-detection. CRUCIAL: region-driven tables are CANDIDATES, not winners.
+    # ``_region_for_caption``'s bbox is reliable for VERTICAL anchoring but often
+    # too NARROW horizontally for a wide table (its width tracks the caption /
+    # partial content, not the full grid), so a naive "region wins" would collapse
+    # a 7-column demographics table to 2 columns. We therefore pick, per caption,
+    # the BETTER of the region-driven candidate and the auto-detected candidate
+    # (``_pick_better_table``: more columns wins; tie-break on cell count) — so
+    # region-driving can only ADD coverage (0×0 stub → real table) or replace an
+    # auto-detect table it strictly beats, never degrade one.
+    layout_doc = _layout_doc
+    region_tables: dict[int, Table] = {}
+    if not camelot_disabled and table_captions:
+        region_tables, layout_doc = _region_driven_capture(
+            pdf_bytes, rejoined, table_captions, layout_doc,
+            next_boundary_by_id, method_pieces,
+        )
+
+    # Auto-detect candidate per caption (legacy pairing). Filter Camelot's output
+    # to tables that have a same-page caption — this anchors detection to caption
+    # signal (matching the pre-pdfplumber-removal behavior) and drops false
+    # positives like bibliographies. Uncaptioned auto-detect tables are dropped as
+    # before (rare in the APA corpus; existing tests are caption-anchored).
     pages_with_table_caption = {c.page for c in table_captions}
+    auto_by_cap: dict[int, Table] = {}
     for ct in camelot_tables:
         if (ct.get("page") or 0) not in pages_with_table_caption:
             continue
@@ -196,7 +218,37 @@ def extract_pdf_structured(
             ct["caption"] = _extract_caption_text(
                 rejoined, match, next_boundary_by_id.get(id(match))
             )
-            tables.append(ct)
+            auto_by_cap[id(match)] = ct
+
+    # Pick the candidate per caption. ``used_caption_ids`` already holds the
+    # auto-detect-matched caps; add any caption the region pass alone satisfied.
+    #
+    # ``_pick_better_table`` is the single arbiter: region-driving's value is that
+    # it gives a STARVED caption its own table — and a starved caption has NO
+    # auto-detect candidate (auto-detect captured the page as one table and
+    # token-paired it to the OTHER caption), so ``auto_td is None`` and region
+    # wins automatically. Where auto-detect DID give this caption a real grid, the
+    # column-collapse guard keeps the richer one. This handles both stacked-table
+    # separation (cog_emo Table 9: auto=None → region 5×3 vs baseline 0×0) and
+    # wide single tables (jama_open_1 Table 1: region too narrow → keep auto 43×7)
+    # without a page-multiplicity special case (which over-forced region on
+    # multi-caption pages whose tables auto-detect already separated cleanly —
+    # bmc_med_3, ieee_access_8 — collapsing their columns).
+    region_win = region_only = 0
+    for cap in table_captions:
+        auto_td = auto_by_cap.get(id(cap))
+        region_td = region_tables.get(id(cap))
+        if auto_td is None and region_td is None:
+            continue
+        best = _pick_better_table(region_td, auto_td)
+        if best is region_td and auto_td is not None:
+            region_win += 1
+        elif best is region_td:
+            region_only += 1
+        tables.append(best)
+        used_caption_ids.add(id(cap))
+    if region_win or region_only:
+        method_pieces.append(f"region_pick:{region_win}+{region_only}")
 
     # §A R1 / B1 (NORMALIZATION_VERSION 1.9.23, 2026-05-23): whitespace_cells
     # fallback for caption-detected tables that Camelot could not recover.
@@ -209,8 +261,8 @@ def extract_pdf_structured(
     #
     # Lazy-import the layout extraction + whitespace helper so the rest of
     # the pipeline works in environments without pdfplumber-layout deps; any
-    # failure here falls back transparently to the isolated path.
-    layout_doc = _layout_doc  # reuse caller-supplied doc when available
+    # failure here falls back transparently to the isolated path. `layout_doc`
+    # may already be materialized by the region-driven pass above; reuse it.
     _whitespace_cells = None
     _region_for_caption_fn = None
     unmatched_caps = [
@@ -346,6 +398,176 @@ def _join_split_captions(text: str) -> str:
     return text
 
 
+# A region's bottom is clipped this many points ABOVE the next same-page
+# caption's top, so a short table's region does not extend down into the NEXT
+# table (cog_emo p13: Table 8's matrix region would otherwise absorb Table 9).
+_REGION_NEXT_CAPTION_GAP_PT: float = 2.0
+
+# A region's top is clamped to the caption's top minus this margin (enough to
+# include the caption line's own glyph ascent, but not the content above it), so
+# a region whose SEARCH_ABOVE fallback reached up into a preceding table is
+# pulled back to start at its own caption.
+_REGION_TOP_MARGIN_PT: float = 4.0
+
+
+def _table_cell_count(td: Optional[Table]) -> int:
+    if not td:
+        return 0
+    return sum(1 for c in (td.get("cells") or []) if (c.get("text") or "").strip())
+
+
+def _pick_better_table(
+    region_td: Optional[Table], auto_td: Optional[Table]
+) -> Optional[Table]:
+    """Choose between a caption's region-driven candidate and its auto-detected
+    candidate. Region-driving anchors the RIGHT table to the caption but its
+    bbox can be too narrow (collapsing columns), so it only wins when it is at
+    least as wide as the auto-detect table AND carries comparable content.
+
+    Rules (auto-detect is the incumbent; region must clearly not regress it):
+      - exactly one present → that one.
+      - region has FEWER columns than auto → auto (guards the column-collapse
+        regression: a 7-col demographics table must not become 2-col).
+      - region has MORE columns than auto → region, UNLESS it lost a large share
+        of cells (a wider-but-emptier grid is a mis-detection, keep auto).
+      - equal columns → whichever has more populated cells (ties → auto, the
+        stable incumbent).
+    """
+    if region_td is None:
+        return auto_td
+    if auto_td is None:
+        return region_td
+    rc = int(region_td.get("n_cols") or 0)
+    ac = int(auto_td.get("n_cols") or 0)
+    r_cells = _table_cell_count(region_td)
+    a_cells = _table_cell_count(auto_td)
+    if rc < ac:
+        return auto_td
+    if rc > ac:
+        # Wider grid must still retain most of the content to be believed.
+        if a_cells > 0 and r_cells < 0.6 * a_cells:
+            return auto_td
+        return region_td
+    # Equal column count: prefer the fuller grid; tie goes to the incumbent.
+    return region_td if r_cells > a_cells else auto_td
+
+
+def _region_driven_capture(
+    pdf_bytes: bytes,
+    rejoined: str,
+    table_captions: list[CaptionMatch],
+    layout_doc,
+    next_boundary_by_id: dict[int, Optional[int]],
+    method_pieces: list[str],
+) -> tuple[dict[int, Table], object]:
+    """Drive Camelot once per caption with the caption-anchored region as
+    ``table_areas`` and return ``{id(cap): Table}`` for every caption that
+    yielded a usable table, plus the (possibly newly-materialized) layout doc.
+
+    For each caption we (1) locate its layout region via ``_region_for_caption``,
+    (2) clip the region bottom at the next same-page caption so it can't reach
+    into the following table, (3) convert the pdfplumber TOP-DOWN bbox to the
+    Camelot PDF BOTTOM-UP ``table_areas`` string, and (4) attach the extracted
+    caption text. Any failure degrades gracefully to ``{}`` so the legacy
+    auto-detect + whitespace path still runs.
+    """
+    empty: dict[int, Table] = {}
+    try:
+        from .tables.detect import _region_for_caption, _bbox_of_caption_line
+        from .tables.camelot_extract import extract_tables_camelot_by_region
+        if layout_doc is None:
+            from .extract_layout import extract_pdf_layout
+            layout_doc = extract_pdf_layout(pdf_bytes)
+    except Exception as exc:
+        record_fallback("region_driven_setup_exception", detail=type(exc).__name__)
+        return empty, layout_doc
+
+    pages = getattr(layout_doc, "pages", None) or ()
+
+    # Group captions per page (top-to-bottom) so the next-caption bottom-clip can
+    # look at the following caption on the same page.
+    by_page: dict[int, list[CaptionMatch]] = {}
+    for c in table_captions:
+        by_page.setdefault(c.page, []).append(c)
+
+    specs: list[dict] = []
+    cap_by_key: dict[str, CaptionMatch] = {}
+    for page, caps in by_page.items():
+        if not (1 <= page <= len(pages)):
+            continue
+        page_obj = pages[page - 1]
+        page_height = float(getattr(page_obj, "height", 0.0) or 0.0)
+        if page_height <= 0.0:
+            continue
+        caps_sorted = sorted(caps, key=lambda c: c.char_start)
+        cap_top_td: dict[int, Optional[float]] = {}
+        for c in caps_sorted:
+            cb = _bbox_of_caption_line(page_obj, c)
+            cap_top_td[id(c)] = cb[1] if cb else None
+        for i, c in enumerate(caps_sorted):
+            try:
+                region = _region_for_caption(layout_doc, c)
+            except Exception as exc:
+                record_fallback("region_driven_region_exception", detail=type(exc).__name__)
+                region = None
+            if region is None:
+                continue
+            x0, top, x1, bottom = region.bbox
+            # Clip the region TOP at the caption's own top. A table never extends
+            # ABOVE its caption line; ``_region_for_caption``'s SEARCH_ABOVE
+            # fallback can pull the region up into a PRECEDING table when no
+            # geometry is found below (cog_emo Table 9, whose caption sits below
+            # Table 8's matrix, otherwise captures Table 8's rows). Clamp so the
+            # region starts at the caption line (minus a small margin for the
+            # caption glyphs' own ascent), never higher.
+            this_cap_top = cap_top_td.get(id(c))
+            if this_cap_top is not None and top < this_cap_top - _REGION_TOP_MARGIN_PT:
+                top = this_cap_top - _REGION_TOP_MARGIN_PT
+            # Clip bottom at the next same-page caption that sits below this one.
+            for j in range(i + 1, len(caps_sorted)):
+                nxt_top = cap_top_td.get(id(caps_sorted[j]))
+                if nxt_top is not None and nxt_top > top + 5.0:
+                    bottom = min(bottom, nxt_top - _REGION_NEXT_CAPTION_GAP_PT)
+                    break
+            if bottom <= top:
+                continue
+            key = f"{id(c)}"
+            specs.append({
+                "key": key,
+                "page": page,
+                "area": f"{x0:.2f},{page_height - top:.2f},{x1:.2f},{page_height - bottom:.2f}",
+                "area_bu": (x0, page_height - top, x1, page_height - bottom),
+                "label": c.label,
+            })
+            cap_by_key[key] = c
+
+    if not specs:
+        return empty, layout_doc
+
+    try:
+        by_key = extract_tables_camelot_by_region(pdf_bytes, specs)
+    except Exception as exc:
+        record_fallback("region_driven_extract_exception", detail=type(exc).__name__)
+        return empty, layout_doc
+
+    result: dict[int, Table] = {}
+    for key, td in by_key.items():
+        cap = cap_by_key.get(key)
+        if cap is None:
+            continue
+        # Use the caption-number id scheme (``t{number}``) — the same the
+        # isolated/whitespace path uses — so ids stay deterministic and match the
+        # recognized-prefix invariant (test_table_ids_unique_and_sequential).
+        td["id"] = f"t{cap.number}"
+        td["caption"] = _extract_caption_text(
+            rejoined, cap, next_boundary_by_id.get(id(cap))
+        )
+        result[id(cap)] = td
+    if result:
+        method_pieces.append("region_driven")
+    return result, layout_doc
+
+
 def _find_caption_for_table(
     camelot_table: Table,
     captions: list[CaptionMatch],