docpluck 2.4.96__tar.gz → 2.4.98__tar.gz

{docpluck-2.4.96 → docpluck-2.4.98}/.claude/skills/_project/lessons.md

@@ -641,3 +641,15 @@ Rotation picks `pool[(N mod L) : (N mod L) + rotation_size]` wrapping. Over `cei
 **What (fix):** `api/cron/daily-digest` intermittently logged "Daily dispatch failed" (one-error admin digest email). Ground truth came from prod `system_logs.context.errorStack` (the logger persists `errorName`/`errorMessage`/`errorStack`; queried read-only via the Neon `DATABASE_URL` in `PDFextractor/frontend/.env.local`): two failed runs (06-17, 06-18) were transient **Neon serverless** errors — `terminating connection due to administrator command` (connection recycled on scale-to-zero) and a control-plane HTTP 500 tagged `"neon:retryable":true` — thrown at DIFFERENT queries, with adjacent days succeeding. NOT a logic bug: both unattended crons fire ~15 sequential neon-http calls at 03:00/09:00 UTC against likely-cold compute and the driver does no retries, so one blip aborts the whole run (and Vercel Cron does not re-run failed crons). Fix: new `frontend/src/lib/db-retry.ts` (`withDbRetry`/`isRetryableDbError` — 3 attempts, 250/500ms backoff, retries ONLY transient/connection-class signatures, never app errors) wrapping every DB call in `notifications/dispatch.ts` + `admin/blob-cleanup/route.ts` + the in-band `sendEmail` (safe to replay — `email_events` ON CONFLICT de-dupes). Regression test `frontend/src/lib/db-retry.test.ts` via `node --test` (Node 26 strips TS natively → zero new deps; first frontend unit test + `npm test` script; excluded from `next build` via tsconfig). Shipped commit `ea86e85`; post-deploy `?dryRun=1` smoke = HTTP 200 + clean DispatchResult.
 
 **How to detect next time:** (1) The admin digest aggregates the PREVIOUS 24h, so a "1 error" email naming `api/cron/daily-digest` is usually the prior day's transient blip, not a live outage — before theorizing, query `system_logs` for the real `errorName`/`errorStack` (it's `NeonDbError`, often `neon:retryable:true`). (2) ANY unattended Vercel cron doing a burst of neon-http queries needs `withDbRetry`: Neon scale-to-zero WILL recycle a connection mid-run and there is no driver-level retry. (3) `next build` here does NOT run eslint (Next 16 — see card `nextjs-16-build-does-not-run-eslint`), so a red `npm run lint` does not block deploy; keep the lint gate green separately for `/ship`.
+
+## 2026-06-25 — Branch can lag its own pushed tags; `check_app_pin_sync.py` PASS does NOT prove the branch is current
+
+**Symptom.** Session opened "clean", `check_app_pin_sync.py` PASS — yet `main` (local AND `origin/main`) was at the v2.4.95 commit while tags `v2.4.96`/`v2.4.97` were already pushed 4 commits ahead, the app pin had auto-bumped to `@v2.4.97`, and the working tree `__version__` read 2.4.95. So **production ran v2.4.97 while the library mainline + working tree claimed v2.4.95.** A prior session tagged+pushed without fast-forwarding the branch, and the tree was checked out backwards.
+
+**Why the gate didn't catch it.** `check_app_pin_sync.py` compares the app pin to the *latest tag* — both were v2.4.97, so PASS. It does not compare the *branch HEAD* to the latest tag. And its working-tree note had a direction bug: it told the BEHIND tree it was "ahead — tag+push v2.4.95" (would have tagged an OLDER version). Fixed the note (ordered `_vtuple()`, commit `59cff5b`); the branch-vs-tag check is still a manual step.
+
+**Rule.** At session start (and before any release), run `git tag --sort=-v:refname | head` and `git rev-parse main origin/main` and compare to the latest `v*` tag. If `main` is an ANCESTOR of the latest tag → the branch lags its own release; fast-forward (`git merge --ff-only v<latest>`) before iterating, or you'll work on a stale base and risk re-doing / colliding with shipped work. A green `check_app_pin_sync.py` is necessary but NOT sufficient evidence the branch is current. Companion: card `release-version-collision-with-parallel-uncommitted-stream`.
+
+## 2026-06-25 — Word-multiset preservation is BLIND to reading-order regressions; AI-verify is mandatory before any reorder flip
+
+The RC-1 banded column-correction flag passed a full-corpus word-preservation scan (26/26 baseline papers, 0 multiset violations) — looked safe to flip default-ON. The 8-canary AI-verify then found **3 ON_REGRESSION** the scan was structurally blind to: running-header furniture injected into prose, Abstract/Intro section order inverted, prose fragmented on a *single-column* paper (false-positive gutter). A pure reorder/furniture-injection preserves the word multiset exactly. **Never flip a reading-order-affecting default on word-preservation (or char-ratio / Jaccard) evidence alone — those gates cannot see "right words, wrong order / wrong place." AI-verify against the gold is required, and the bar is zero ON_REGRESSION across the full canary set.** Re-validates the project ground-truth hard rule.

{docpluck-2.4.96 → docpluck-2.4.98}/.claude/skills/docpluck-iterate/LEARNINGS.md

@@ -1327,3 +1327,44 @@ aren't skipped.
 ### Verification Gaps / Deferred (queued, not dropped)
 - **RC-T Layer-1 recovery** — actually RECOVERING lost table data via tight `table_areas` (plos_med T5's 13 SAE rows; chan_feldman T2 column-squish) — out of Option-A scope, deferred by the user.
 - **Audit of the ~37 `_strip_phantom` th-stripped tables** — some title-shaped `<th>` strips may be wrongly-stripped REAL tables (pre-existing, predates this cycle). Needs its own verification cycle (render each, judge real-vs-phantom). Surfaced explicitly; NOT silently shipped around.
+
+---
+
+## Run: 2026-06-25 · cycle 1 (investigation + verification) · @ v2.4.97 · verdict PARTIAL
+
+**Goal:** user invoked `/docpluck-iterate`; established stop-condition implicitly via the two scope decisions below. Targeted RC-T table-bbox (then, on evidence, the RC-1 banded flip).
+
+### What landed (clean, shipped, verified)
+1. **Branch reconciliation (the big one).** At session start `main` (local AND `origin/main`) was at the v2.4.95-era commit `2dbdd98`, but tags `v2.4.96`/`v2.4.97` were already pushed 4 commits AHEAD (real, test-backed RC-T + DP-2/DP-5 work) and the app pin had auto-bumped to `@v2.4.97` → **production ran v2.4.97 while the library mainline claimed v2.4.95.** Fast-forwarded `main` (local + origin) to `813aa4c` (v2.4.97); now mainline == latest tag == app pin. **Lesson: at session start, ALWAYS check `git tag` vs `git rev-parse main origin/main` — a prior session can push tags without advancing the branch, and the working tree can be checked out backwards. The `check_app_pin_sync.py` PASS hid it (pin==tag is true even when the *branch* lags the tag).**
+2. **`check_app_pin_sync.py` direction bug + test** (commit `59cff5b`). `compare()` said "working-tree X is ahead of tag Y — tag+push X" for ANY `X!=Y`; a tree BEHIND its tag (2.4.95 < 2.4.97, exactly this session) got told to tag an OLDER version. Fixed with ordered `_vtuple()`; `tests/test_check_app_pin_sync.py` (10 cases, fails-before/passes-after).
+
+### What was investigated and correctly NOT shipped
+3. **RC-T table-data recovery = the RC-1 banded layout-channel work (architectural).** ip_feldman T10 (and 3 other canary orphans) lose real table data; gold confirms a 7-row table, render drops all 7. Probed FOUR recovery angles (Camelot free-form, whitespace_cells full region, banded text-channel flag, band-clipped whitespace_cells) — each bottoms out in: full-page Camelot bbox / region-overshoot-into-prose / RC-1 interleave INSIDE the table region (table rows + Discussion prose share y-positions) / tight-kerned glued stat cells that defeat x-gap column detection (`pdfplumber_extract_words_unreliable`). The orphan `### Table N` heading is the DELIBERATE v2.4.55 clean-fail (keeps caption for `table_parity`), NOT the bug. **Lesson: reproduce + probe before trusting a "small fix" hypothesis — my B2-is-easy guess was disproven in ~15 min of probing, saving a mis-scoped cycle.** (card `reproduce-triage-defect-at-head`)
+4. **RC-1 banded default-flip ATTEMPTED, REJECTED by 8-canary AI-verify.** Word-preservation corpus scan = 26/26 PRESERVED, 0 violations (looked safe!). But the 8-canary AI-verify (8 parallel Sonnet, Claude Max) found **3 ON_REGRESSION** (deterministically confirmed): `ar_apa` flag-ON injects `M. Muraven / Journal` running-header furniture ×2 + inverts Abstract/Intro order; `chan_feldman` injects Power-Analysis prose into Extension; `maier` (SINGLE-COLUMN) fragments prose via a false-positive gutter. **THE LESSON (re-validates the project hard rule): the word-multiset gate is BLIND to reordering + furniture-injection — it passed 26/26 while the flag corrupted 3 canaries. AI-verify is necessary, not optional, before any reading-order default flip.** Flag stays default-OFF. Precise gap-5/6/7 blockers + a stricter zero-regression re-verify gate written to the RC-1 spec.
+
+### Verification-infra defect re-confirmed (surfaced, not fixed — needs portfolio decision)
+5. **The pre-push canary-audit.sh hook clobbered run-meta `phase_5d_runs` + `cycle_status` on BOTH pushes this session** — overwriting my honest FAIL verdicts with `AUDIT_DEFERRED_TO_AGENT → union PASS` placeholders (`cycle_status` flipped to PASS, a broken corpus reading green). Restored truthful verdicts after each push per memory `feedback_canary_audit_clobbers_phase5d`. **This bug actively manufactures false-green on a FAIL corpus; it bit twice in one session.** The fix (make the hook not write deferred-PASS, or write BLOCKED/AUDIT_DEFERRED instead of PASS) is a substrate change in `~/.claude/skills/_shared/iterate-loop/canary-audit.sh` — surfaced for a dedicated substrate cycle.
+
+### Open (queued, NOT dropped — rule 0e-bis)
+- 4 canary FAILs remain (ip_feldman T10, chan_feldman T6/T9, chandrashekar T2 table-data loss; plos_med sidebar interleave) → the RC-1 banded layout-channel + sidebar-interleave architectural cycle (spec `2026-06-08-rc1-region-aware-column-architecture.md`, sharpened this run).
+- Run verdict: **PARTIAL** — 2 clean fixes shipped; the canary corpus is NOT clean (honest punch-list above), so the standing corpus verdict is FAIL and the iterate-gate `--close` correctly cannot green-close.
+
+---
+
+## Run: 2026-06-25 (cont.) · RC-T char-level column recovery — FOUNDATION landed (v2.4.98), T10 wiring deferred
+
+### Outcome
+- **char_whitespace_cells (the char-level absolute-x-gap column detector) IMPLEMENTED + tested** in `docpluck/tables/whitespace.py`, wired as an automatic fallback inside `whitespace_cells` when the word path finds < 2 columns. This is the layout-channel piece that finally makes the "RC-T table-data recovery" tractable.
+
+### Blind spot closed (the key technical finding)
+- **Earlier this session I assessed T10 recovery as "architectural / multi-session, char detection too hard."** That was WRONG — disproven by probing char-level x-positions: ip_feldman Table 10's 3 stat columns are cleanly separable at the CHAR level (gaps 23-78pt) even though pdfplumber's WORD grouper glues the whole numeric row (`.29***−.21***.07`) into one token so the word-gap detector finds nothing. This is exactly the `pdfplumber_extract_words_unreliable` lesson ("always carry a char-level absolute-x-gap fallback") — I should have reached for it before declaring the wall.
+- **Right-aligned numeric tables need column-START-edge voting, not gap-MIDPOINT voting.** The label column is variable-width (`Loneliness` vs `Social orientation scale`), so gap midpoints scatter and never reach the stability threshold; but the DATA columns are left-aligned to fixed x, so their left edges are stable. `_find_stable_column_boundaries(bucket_pt>0)` votes on column starts + buckets within 8pt. With this, char_whitespace_cells recovers all 7 T10 data rows matching the gold exactly.
+
+### Edge case
+- **The word path MUST stay byte-identical when adding a char path.** First cut refactored `_find_stable_column_boundaries` to count only multi-column rows in the denominator — that silently changed the WORD path's stability threshold and broke 2 existing table tests (caught at ~65% of the broad run). Fix: branch the function — `bucket_pt<=0` runs the verbatim original word logic, `bucket_pt>0` runs the new char logic. Lesson: when extending a shared helper, keep the existing caller's path provably unchanged (literal copy), don't "improve" it in the same edit.
+
+### Deferred (queued, NOT dropped — rule 0e-bis)
+- **char_whitespace_cells alone does NOT close the T10 canary in production.** T10's caption is *matched* by the garbage full-page Camelot table, so it's added directly to `tables` and the whitespace fallback (where the char path lives) is never reached for it. Closing T10 needs the **degenerate-Camelot-table replacement** in `extract_structured.py`: when a Camelot-matched table has a full-page-ish bbox + furniture/prose cells, discard it and re-extract via `whitespace_cells` on the gutter-band-clipped caption region, PLUS a region prose-trim (the `(extension)` wrapped-caption + Discussion-prose tail still contaminate the region). That change touches the core extraction path for ALL papers → requires the full ~48-paper guard-live-vs-bypassed diff + 7-canary AI-verify before it ships (cycle-3 caption-follows revert is the precedent). Foundation is in place; this is the next focused cycle.
+
+### Verification
+- `tests/test_whitespace_char_fallback.py` (5 cases incl. real-PDF ip_feldman T10 = all 7 gold rows recovered; synthetic tight-kerned recovery; word-space reinsertion; delegation wiring; single-column-no-fabrication guard). 15/15 pass on the whitespace/char/caption-table suite; 0 failures across the broad table/render run; word path proven byte-identical (restored verbatim).

{docpluck-2.4.96 → docpluck-2.4.98}/CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## [2.4.98] — 2026-06-25
+
+**ESCIcheck handoff defects (DOCPLUCK_HANDOFF_2026-06-25): η²p effect-typing, inline correlation r-typing, and bracketed-CI continuation merge — flatten/cell-cleaning only, AI-gold-verified.** `TABLE_EXTRACTION_VERSION` → `2.4.4`. Render-visible in the inline flattened-table blocks + `.tables.jsonl` sidecar `fields`. **No capture-path change** — caption→table pairing is byte-identical to v2.4.3. Grounded against the AI `reading`/`stats` golds (article-finder), not pdftotext, and confirmed by a 6-paper AI-gold canary verify. Triage: `docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md`.
+
+- **DP-3 (table side) — type a font-dropped partial-η² effect column.** Many APA ANOVA tables report effect size as `η²p`, but the glyph lives in a font with no ToUnicode mapping (`pdffonts`: `uni:no`), so pdftotext AND pdfplumber both decode it as a space — the effect-size column header is blank and no "eta"/"η" token survives anywhere, so `tables.flatten._effect_key` fell back to the generic `est` key and the consumer got a nameless number. `flatten._infer_anova_eta2_hint` now types an unlabeled estimate column as `eta2` when the table is structurally an F-test/ANOVA results table (carries an `F` column + a `BF01` and/or CI column) and names NO competing effect (`d`/`dz`/`r`/`OR`/`g`); the value is range-guarded to η²'s domain `[0, 1]`. Keyed on the structural signature, never paper identity. `collabra.90203` Tables 8 & 9 now emit `eta2` matching the gold's `η²p` column (`.06, .00, .01, .04, …`). **The text-channel η²p (in body prose) is unrecoverable — OCR-tier won't-fix** (glyph identity absent from the PDF), as is DP-6 figure-inset text.
+
+- **DP-5 (b) — type inline self-labeled correlation cells.** A cell that states its own statistic — `r = .67`, `r = -.73`, `d = 0.32` — is now typed by that token even when the column header is generic ("Effect size") and the column-role recovery produced no estimate for it (`flatten._inline_stat_field`). `cog_emo` (`10.1080/02699931.2024.2434156`) Table 8's intercorrelation rows now carry a typed `r` with their CI/p, matching the gold. The display value is the number only, so the sentence renders `r = 0.63` (not `r = r = .67`).
+
+- **DP-5 (a) — merge a CI split across two rows by its bracket tail.** Camelot stream splits a wrapped CI cell — `[0.59,` on one row, `0.73]` on the next. `cell_cleaning._merge_continuation_rows` already folded the *parenthetical* tail (`8.34)`) but not the *bracketed* CI tail (`0.73]`); `_is_fragment_cell` now recognizes a close-bracket tail, so the bound rejoins (`[0.59, 0.73]`) and the bare-fragment junk row no longer survives (cog_emo Table 8: 14 → 10 rows, CIs complete).
+
+**Deferred (prototyped + REVERTED this cycle — surfaced, not dropped): DP-1 / DP-2 capture recovery.** A caption page-attribution fix (`find_caption_matches` advancing `char_start` past the leading `\f` its `^\s*` ate) does correct the off-by-one that mis-pages a table starting a page, and in isolation it unblocked `collabra.77859` Table 4's replication `t/df/d` (DP-1). But the **mandatory AI-gold canary verify showed it is net-harmful corpus-wide**: populating the previously-empty `line_text` re-scores `_find_caption_for_table`'s same-page token overlap and surfaces low-quality whitespace tables, mis-pairing tables whose captions share a page (efendic T4/T5, cog_emo T8/T9 swapped) and only half-fixing plos_med (Table 4 still wrong, Table 5 degraded). Reverted to keep the corpus regression-free. It needs same-page-caption disambiguation + whitespace-region quality gating, queued as its own gated cycle (full 48-paper guard-diff + canary AI-verify). DP-2's Table 1 (an 8-column grouped table) is beyond the lineless fallback regardless. See the triage doc.
+
+**Also fixed (pre-existing, leave-nothing-behind):** `test_table_html_renders_when_structured` and `test_table_kinds_are_valid` asserted a non-structured table must be `kind == "isolated"`, but the layout-channel `whitespace` fallback (a valid kind since the v2.4.3 foundation) can surface — including when Camelot returns "no tables" under cumulative test-suite load (memory `feedback_camelot_flake_cumulative_load`). The tests now accept `whitespace` (asserting cells + HTML present, no Camelot confidence), so a Camelot load-flake on the jama/ieee/chen fixtures no longer mis-fails them.
+
+Verification: shipped flatten/cell-cleaning changes covered by `tests/test_tables_flatten.py` (TestInlineSelfLabeledStat ×3, TestBracketTailCIContinuation) + `tests/test_tables_flatten_blank_header_recovery.py::TestT90203Table8BlankHeaders` (eta2 typed, est gone). Full library suite green under `pytest -n10` (real-PDF Camelot tests flake only under serial cumulative load — each passes per-file / under xdist). 6-paper AI-gold canary verify confirms DP-3 (collabra.90203 η²p), DP-5 (cog_emo r + CI) correct; it is what caught the DP-1/DP-2 page-fix regression and drove its revert.
+
+---
+
+**RC-T foundation (earlier same-day commit): char-level column-detection fallback for tight-kerned tables.** `TABLE_EXTRACTION_VERSION` → `2.4.3` (now → `2.4.4` per above). Library-internal capability addition — a foundation for the still-open RC-T Layer-1 table-data recovery.
+
+- **`docpluck/tables/whitespace.py` gains `char_whitespace_cells`** — a char-level absolute-x-gap column detector, wired as an automatic fallback inside `whitespace_cells` that fires ONLY when the word-gap detector finds < 2 columns. On tight-kerned PDFs pdfplumber's word grouper glues a whole numeric row into one token (e.g. ip_feldman Table 10's `.29***−.21***.07`), so the word path finds no column gaps; the chars are still cleanly separated by large inter-column gaps. The fallback recovers the grid from char x-gaps, **voting on column-START edges** (in a right-aligned numeric table the label column is variable-width so gap midpoints scatter, but the data columns are left-aligned to fixed x), with an 8pt boundary bucket, and reinserts intra-cell word spacing from glyph geometry.
+- **Strictly additive — currently-correct tables are byte-unchanged.** The word path (`_find_stable_column_boundaries(bucket_pt=0)`) is restored verbatim to its pre-change behavior; only a region that previously yielded NO whitespace grid (`whitespace_cells` returned `[]`) can now gain one, and only when char-columns clear a 60%-row stability bar. In isolation this recovers all 7 of ip_feldman Table 10's regression-coefficient rows matching the AI gold.
+
+**Not yet closed (next gated cycle):** the char fallback alone does NOT fix Table 10 in the live pipeline — T10's caption is *matched* by a degenerate full-page Camelot table, which is used directly, so the whitespace fallback (where the char path lives) is never reached. Closing it needs an `extract_structured.py` change — detect a degenerate matched Camelot table (full-page-ish bbox + furniture/prose cells), discard it, and re-extract via `whitespace_cells` on the gutter-band-clipped caption region, plus a region prose-trim — which touches the core extraction path for all papers and so requires the full ~48-paper guard-diff + 7-canary AI-verify before it ships. See `LEARNINGS.md` 2026-06-25 and `docs/superpowers/specs/2026-06-21-rc-t-table-region-prose-contamination.md`.
+
+Verification: `tests/test_whitespace_char_fallback.py` (5 cases incl. real-PDF ip_feldman Table 10 = all 7 gold rows recovered, synthetic tight-kerned recovery, intra-cell word-space reinsertion, delegation wiring, single-column-no-fabrication guard); 15/15 pass on the whitespace/char/caption-table suite across two independent runs; 0 failures across the broad table/render suite; word path proven byte-identical.
+
+## [2.4.97] — 2026-06-22
+
+**Three table fixes shipped together (combined from two concurrent sessions): type the skipped p+df columns (DP-2), stop dropping / mis-binding two-header-row tables (DP-5), and stop the table raw_text fallback swallowing body prose (RC-T Layer-2).** `TABLE_EXTRACTION_VERSION` → `2.4.2`; no `NORMALIZATION_VERSION` / `SECTIONING_VERSION` change. DP-2/DP-5 are render-visible in the inline flattened-table blocks + the `.tables.jsonl` sidecar `fields` (the `<table>` HTML gains the previously-dropped data rows); RC-T Layer-2 is render-visible in the `unstructured-table` fallback blocks. DP-2/DP-5 filed in `ESCIcheckapp/docs/DOCPLUCK_HANDOFF_2026-06-21.md`; RC-T Layer-2 per `docs/superpowers/specs/2026-06-21-rc-t-table-region-prose-contamination.md`.
+
+- **DP-2 — type the unlabeled p and df columns.** `tables.flatten._recover_blank_roles` recovered the leading test statistic and the `d [CI]` column of a header-stripped result table but left the bare p-value and df columns between them untyped, so `collabra.77859` Table 3 emitted `fields: {group, t, d, CI}` and dropped the `p` (`.551`) and `df` (`260.54`). A new Pass 4.5 types a still-blank column that is a bare `.XXX` with no comparison op as `p`, and a bare integer / Welch-decimal sitting between the test statistic and its `est/CI` column as `df` — keyed on data shape + position relative to the already-recovered roles, never bare position. The four Table-3 rows now carry `p` and `df`.
+
+- **DP-5 — two-row-header parallel-arm tables: recover the first data row and align centered super-headers.** `collabra.90203` Table 10 delivered only 5 of its 6 correlation rows (the Identifiable/Explicit-learning row was silently dropped), and the Original/Replication arms of `xiao_2021` Table 4 were swapped. Three coupled root-cause fixes: (a) `cell_cleaning._is_header_like_row` now counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, `N/A`) as data via `_DATA_VALUE_CELL_RE`, so a real first data row is no longer mis-read as a third header row (the bracket branch requires a digit and no letters inside, so a genuine `[95% CI]` header stays a header); (b) `tables.flatten._detect_column_groups` re-derives arm boundaries from equal-width blocks of the data region — each must contain exactly one super-label — so a *centered* super-label (camelot stream loses colspan and folds it mid-span) no longer swaps arm values or pushes a stat column into the label region; left-aligned super-headers stay byte-identical; (c) `tables.flatten._classify_column` reads a folded super-header cell's role from its sub-part so a folded `…<sep>95% CI` column is still typed `CI`. Table 10 now emits all 6 conditions split into Target-article / Replication arms with correct `r` / `n` / `CI` / `p`; xiao Table 4 arms are no longer swapped; incidentally recovers `chan_feldman` Table 8 arm labels and `jama_open_2` Table 3 HR estimates + CIs.
+
+- **RC-T Layer-2 — stop the table raw_text fallback swallowing body prose.** When Camelot recovers no cells, `extract_structured._extract_table_body_text` linearises the text after a caption as the `unstructured-table` fallback; its per-line prose gate (`_line_is_body_prose`, len ≥ 80) misses prose that pdftotext WRAPPED into short (~48-char) lines, so a short table's caption-anchored region overshot the table end and swallowed Results/Discussion prose. Two FP-safe structural fixes: **(a) Note-anchor** — a table's `Note:` footnote is, by convention, its last element, so trim everything after the note paragraph (`chan_feldman` T1/T3 + `efendic_2022` T5 trailing Discussion prose removed; the stat rows + the note are kept); **(b) degenerate-prose guard** — suppress a fallback block that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so the renderer emits a clean caption-only table (`chan_feldman` T9 was an entire verbatim duplicate of `## Discussion` — now caption-only, no duplication). FP-safe by construction: real table cells start with a header / label / number / single-letter item marker, never a wrapped mid-sentence continuation — hypotheses ("a There is a positive association…"), descriptive rows ("Median age"), and instrument fragments are preserved. Keyed on the structural overshoot signature, never paper identity.
+
+Verification: new real-PDF + contract regression tests (`tests/test_tables_superheader_alignment_real_pdf.py`) — collabra.90203 T10 six-conditions/correct-arms + xiao T4 not-swapped (each FAILS at HEAD, PASSES after), plus `_is_header_like_row` / `_detect_column_groups` contract cases; `tests/test_tables_flatten_blank_header_recovery.py` extended for DP-2. A full-corpus (101-PDF) cached-table flatten diff confirms no clean-table regression — every changed table is a recovered row, a correct arm split, a recovered field, or a removed stat-less spurious row; already-garbage tables shuffle without a clean table regressing. Broad pytest green (real-PDF Camelot tests run serially per file — non-deterministic under cumulative load). RC-T Layer-2 adds `tests/test_rc_t_layer2_raw_text_real_pdf.py` (6 contract + 4 real-PDF: chan T1 Note-anchor, T9 suppress-no-duplication, T3 preserved) and an independent full-corpus 101-PDF guard-live-vs-bypassed raw_text diff (`grew=0 changed=0`; 4 trims + 8 prose-suppressions only). A 7-canary Sonnet AI-gold verify confirms every table this release touched is correct (chan T1/T3/T9, maier T10 six-conditions, xiao T4 arms) with no new TEXT-LOSS / HALLUCINATION.
+
+**Deferred (pre-existing, user decision 2026-06-22):** the remaining canary AI-verify FAILs are the architectural backlog, NOT regressions from this release — RC-T **Layer-1** table-data recovery (`table_areas`; e.g. plos_med Table 5's SAE rows, chan_feldman / chandrashekar under-extraction) and RC-1 two-column / sidebar column-interleave. Tracked in `docs/TRIAGE_2026-06-21_head_v2.4.95_assessment.md`; intentionally not addressed here.
+
 ## [2.4.96] — 2026-06-21
 
 **RC-T (Option A): strip Camelot "tables" that are absorbed body prose, not data.** Render-only — `render.py::_strip_phantom_camelot_tables`; no `TABLE_EXTRACTION_VERSION` / `NORMALIZATION_VERSION` / `SECTIONING_VERSION` change.

{docpluck-2.4.96 → docpluck-2.4.98}/LESSONS.md

@@ -274,6 +274,74 @@ Cite: `docpluck/tables/camelot_extract.py::_augment_lattice_with_stream_rows` +
 
 ---
 
+## L-010 — A caption that starts a page is mis-paged by the `^\s*` form-feed; and a font with no ToUnicode makes its glyph unrecoverable (recover the column ROLE, not the glyph)
+
+### Two findings, both surfaced 2026-06-25 by the ESCIcheck handoff
+
+1. **The `^\s*` caption regex eats the `\f` and mis-pages a page-starting table.**
+   `TABLE_CAPTION_RE` / `FIGURE_CAPTION_RE` begin `^\s*`. When a table starts a new
+   page, pdftotext emits `…results\n\fTable 4. …`, and `\s*` consumes the `\f`, so
+   `m.start()` lands *before* the form-feed. `_page_for_offset` then counts the
+   caption on the page BEFORE the break (off-by-one), and `_line_at` returns the
+   empty pre-`\f` segment (so `line_text == ''`). With the wrong page,
+   `_bbox_of_caption_line` can't find the caption in the layout channel → the whole
+   layout-region lookup returns None → the whitespace/char fallback never fires and
+   the table degrades to a caption-only stub. On `collabra.77859` this hit **all 5
+   tables**. The seemingly-obvious fix — advance `char_start` past the leading `\f`
+   to the actual "Table"/"Figure" token (`captions.find_caption_matches`) — DOES
+   correct the page AND unblock `collabra.77859` Table 4's replication stats (DP-1) in
+   isolation. **It was tried 2026-06-25 and REVERTED.** Populating the
+   previously-empty `line_text` re-scores `_find_caption_for_table`'s same-page
+   token-overlap and surfaces low-quality whitespace tables, so the mandatory AI-gold
+   canary verify caught it mis-pairing tables whose captions share a page (efendic
+   T4/T5, cog_emo T8/T9 swapped) and only half-fixing plos_med. **The lesson is the
+   process, not the patch:** a capture-path change that helps one paper in isolation
+   can silently mis-pair others — only a corpus-wide AI-gold verify (NOT the unit
+   suite, which stayed green) reveals it. The real fix needs same-page-caption
+   disambiguation in `_find_caption_for_table` + whitespace-region quality gating
+   FIRST; queued as its own gated cycle. Symptom to watch: `line_text == ''` on a
+   caption, or a caption whose `page` is one less than where the table visibly is.
+
+2. **A glyph with no ToUnicode mapping is gone from BOTH channels — recover its
+   ROLE, not the glyph.** `collabra.90203` reports `η²p`, but `pdffonts` shows the
+   symbol font as `uni: no` (no ToUnicode CMap), so pdftotext AND pdfplumber both
+   decode the glyph as U+0020 — the text reads `( = .000, …)` and the table's
+   effect-column header is blank. This is the same class as the residual deleted-minus
+   (memory `project_docpluck_rc_b7_done_w0h`): the *character identity* is absent from
+   the PDF, so text-channel recovery is **OCR-tier won't-fix**. But in a TABLE the
+   column's *role* is recoverable from structure — an F-test/ANOVA results table that
+   reports a Bayes factor + CI and names no competing effect reports η²p by APA
+   convention — so type the value `eta2` from the structural signature (range-guarded
+   to η²'s `[0,1]` domain) even though the glyph is unrecoverable
+   (`flatten._infer_anova_eta2_hint`). Don't chase the glyph; recover the meaning.
+
+### The rules
+
+1. **A caption-page / capture-path change MUST be AI-gold-verified across the corpus
+   before shipping — the unit suite will not catch a mis-pairing.** The page-fix kept
+   all 1852 unit tests green yet swapped table↔caption pairings on 3 papers. Per the
+   project ground-truth rule, render the canary set and compare TABLES against the AI
+   `reading` golds; revert if any paper regresses. (`^\s*`-anchored scans that skip
+   `\f` ARE the right idea for page attribution, but the downstream `line_text` /
+   same-page-caption scoring must be made robust in the same change, not after.)
+2. **Before "the symbol got stripped", run `pdffonts`.** If the glyph's font is
+   `uni:no`, nothing in the byte stream carries its identity — stop trying to recover
+   the glyph; recover the column's role/meaning from structure, or mark it OCR-tier.
+   (Shipped: `flatten._infer_anova_eta2_hint` types the value `eta2` from the F-test
+   table structure even though the η²p glyph is gone.)
+3. **A self-labeled cell beats its column header.** `r = .67` / `d = 0.32` states its
+   own type; type by the cell token even under a generic "Effect size" header
+   (`flatten._inline_stat_field`, shipped) — store only the numeric part so the
+   sentence assembler doesn't double the prefix (`r = r = .67`).
+
+Cite (SHIPPED v2.4.4): `docpluck/tables/flatten.py::_infer_anova_eta2_hint` +
+`_inline_stat_field`, `docpluck/tables/cell_cleaning.py::_is_fragment_cell` (bracket-CI
+tail). REVERTED (queued): the `captions.find_caption_matches` char_start advance +
+`whitespace._whitespace_grid_is_clean` / `_trim_trailing_prose_rows` gates.
+See `docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md`, CHANGELOG v2.4.98.
+
+---
+
 ## When to add a new lesson here
 
 Add a lesson when:

{docpluck-2.4.96 → docpluck-2.4.98}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.96
+Version: 2.4.98
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://docpluck.app
 Project-URL: Documentation, https://docpluck.app/api-docs

{docpluck-2.4.96 → docpluck-2.4.98}/docpluck/__init__.py

@@ -78,7 +78,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.96"
+__version__ = "2.4.98"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.96 → docpluck-2.4.98}/docpluck/extract_structured.py

@@ -37,7 +37,7 @@ from .tables.render import cells_to_html
 from .telemetry import record_fallback
 
 
-TABLE_EXTRACTION_VERSION = "2.4.0"  # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
+TABLE_EXTRACTION_VERSION = "2.4.4"  # v2.4.4 (ESCIcheck handoff 2026-06-25 — flatten-only, AI-gold-verified): (DP-3) flatten._infer_anova_eta2_hint types a font-dropped η²p effect column by STRUCTURE — an unlabeled estimate column in an F-test/ANOVA results table (F + BF01/CI, no competing d/r/OR) is keyed `eta2`, range-guarded to [0,1]; the η²p glyph itself is unrecoverable (NotoSerif uni:no, OCR-tier). (DP-5b) flatten._inline_stat_field types a self-labeled cell (`r = .67`, `d = 0.32`) by its token even under a generic "Effect size" header. (DP-5a) cell_cleaning._is_fragment_cell recognizes a bracketed-CI close tail (`0.73]`) so a CI split across rows rejoins (`[0.59, 0.73]`) and the junk fragment row is dropped (cog_emo T8 14→10 rows). These three are flatten/cell-cleaning ONLY — no capture-path change, so caption→table pairing is byte-identical to v2.4.3. (DP-1/DP-2 capture recovery — the caption page-attribution fix — was prototyped but REVERTED: AI-gold canary verify showed it mis-pairs same-page-caption tables (efendic T4/T5, cog_emo T8/T9) and only half-fixes plos_med; it is queued as its own gated cycle with same-page disambiguation + region-quality gating. See docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md.) # v2.4.3 (RC-T foundation): tables.whitespace gains a CHAR-LEVEL column-detection fallback (char_whitespace_cells) — when pdfplumber's word grouper glues a tight-kerned numeric row into one token so the word-gap detector finds < 2 columns, recover the grid from char x-gaps, voting on column-START edges (right-aligned data columns are left-edge-stable even when the label column is variable-width) and reinserting intra-cell word spacing from geometry. Fires ONLY as a fallback when the word path returns < 2 columns, so currently-correct tables are byte-unchanged (word path restored verbatim). Recovers ip_feldman Table 10's 7 regression rows in isolation. NOT yet wired to replace a degenerate MATCHED Camelot table (that extract_structured change + region prose-trim is the next gated cycle — see LEARNINGS 2026-06-25). # v2.4.2 (RC-T Layer-2): _extract_table_body_text now (a) Note-anchor — a table's "Note:" footnote is its last element, so trim body prose bled past it (chan_feldman T1/T3, efendic_2022 T5); and (b) degenerate-prose guard — suppress a raw_text fallback that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so render emits a clean caption-only table instead of an unstructured-table dump duplicating Results/Discussion prose (chan_feldman T9 was a verbatim ## Discussion duplicate). FP-safe (real cells start with header/label/number/single-letter marker, never a wrapped continuation); full-corpus 101-PDF guard-diff only trims+suppresses (grew=0 changed=0). # v2.4.1 (DP-2/DP-5): (DP-2) blank-header role recovery now types the unlabeled p-value (a bare `.XXX` after the test stat, no comparison op) and df (a bare integer/Welch-decimal between the stat and the d[CI] column) columns it previously skipped — collabra.77859 T3 fields gain p+df (tables.flatten._recover_blank_roles Pass 4.5). (DP-5) parallel-arm tables with a TWO-ROW header no longer drop their first data row, and a CENTERED super-header is aligned to its arm block instead of its visual-center column: (a) cell_cleaning._is_header_like_row counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, N/A) as data via _DATA_VALUE_CELL_RE so a real first data row isn't read as a 3rd header row (collabra.90203 T10 recovered the Identifiable/Explicit-learning correlation); (b) tables.flatten._detect_column_groups re-derives arm boundaries from equal-width blocks of the data region (each must hold one super-label) so a centered super-label folded mid-span no longer swaps arm values (xiao_2021 T4 Original/Replication F) or pushes a stat column into the label region; (c) tables.flatten._classify_column reads a folded super-header cell's role from its sub-part (collabra.90203 T10 CI). Full-corpus cached-table flatten diff: no clean-table regression. # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
 
 TableTextMode = Literal["raw", "placeholder"]
 
@@ -1306,6 +1306,74 @@ def _line_is_body_prose(line: str) -> bool:
     return stopwords_hit >= 4
 
 
+def _join_wrapped_lines(lines: list[str]) -> list[str]:
+    """Merge pdftotext-wrapped lines into logical paragraphs.
+
+    pdftotext linearizes a flowing prose paragraph into several short
+    (~45-60 char) lines; the per-line ``_line_is_body_prose`` gate
+    (len >= 80) cannot see prose in that wrapped form. Joining a line with
+    the next whenever it does not end on sentence-terminal punctuation
+    reconstructs the paragraph so prose can be measured at paragraph scale.
+    """
+    paras: list[str] = []
+    cur = ""
+    for ln in lines:
+        s = ln.strip()
+        if not s:
+            continue
+        cur = (cur + " " + s).strip() if cur else s
+        if s.endswith((".", "!", "?", ":")):
+            paras.append(cur)
+            cur = ""
+    if cur:
+        paras.append(cur)
+    return paras
+
+
+def _raw_text_is_degenerate_prose(text: str) -> bool:
+    """True if a table raw_text fallback is dominated by flowing body prose.
+
+    RC-T Layer-2 (v2.4.97). When Camelot recovers no cells AND the
+    caption-anchored region has no extractable table text near the caption,
+    the body_start walk lands INSIDE a prose paragraph and the fallback
+    swallows Results/Discussion prose (which is then duplicated under its
+    real section heading). Such a block must be suppressed (render then
+    emits a clean caption-only table) rather than dumped verbatim.
+
+    FP-safe by construction — fires only when BOTH hold:
+      (a) the block STARTS mid-sentence: its first line begins with a
+          lowercase multi-letter continuation word. A real table's
+          linearized cells start with a column header, label, number, or a
+          single-letter item marker (``a``/``b``/``c``) — never a wrapped
+          mid-paragraph continuation like "than empathy. We provided ...".
+      (b) the joined block is majority (>= 60% of chars) sentence-shaped
+          body prose.
+
+    Legitimate degraded tables are preserved: hypotheses ("a There is a
+    positive association ..."), descriptive rows ("Median age (years)"),
+    instrument items ("h et al., 1997)") all fail (a). Keyed purely on the
+    structural overshoot signature, never on paper identity.
+    """
+    lines = [ln for ln in text.split("\n") if ln.strip()]
+    if len(lines) < 4:
+        return False
+    first_tokens = lines[0].split()
+    first_word = first_tokens[0] if first_tokens else ""
+    starts_midsentence = (
+        len(first_word) >= 2
+        and first_word[0].islower()
+        and first_word[0].isalpha()
+    )
+    if not starts_midsentence:
+        return False
+    paragraphs = _join_wrapped_lines(lines)
+    total = sum(len(p) for p in paragraphs)
+    if total == 0:
+        return False
+    prose = sum(len(p) for p in paragraphs if _line_is_body_prose(p))
+    return prose >= 0.6 * total
+
+
 def _extract_table_body_text(
     raw_text: str,
     cap: CaptionMatch,
@@ -1379,6 +1447,31 @@ def _extract_table_body_text(
             break
         kept.append(ln)
 
+    # Note-anchor table-end (RC-T Layer-2, v2.4.97). A table's "Note:" /
+    # "Notes:" footnote is, by academic-table convention, its LAST element.
+    # Any text after the note paragraph is body prose that bled past the
+    # table boundary — the caption-anchored region overshot the table end
+    # and the per-line `_line_is_body_prose` gate (len >= 80) misses prose
+    # that pdftotext WRAPPED into short (~48-char) lines, so it accumulates
+    # here. Trim everything after the note's (possibly wrapped) paragraph.
+    # This is FP-safe: legitimate table cells (hypotheses a/b/c, instrument
+    # items) appear BEFORE the note; nothing legitimate follows it. Keyed on
+    # the structural "Note: ... <sentence end>" signature, never paper
+    # identity. `^Notes?[.:]` requires punctuation so body prose that merely
+    # starts with the word "Note that ..." does not false-trigger.
+    note_idx = next(
+        (i for i, ln in enumerate(kept)
+         if re.match(r"^\s*Notes?[.:]", ln.strip())),
+        None,
+    )
+    if note_idx is not None and not os.environ.get("DOCPLUCK_RCT_L2_BYPASS"):
+        note_end = note_idx
+        for k in range(note_idx, len(kept)):
+            note_end = k
+            if kept[k].strip().endswith((".", "!", "?")):
+                break
+        kept = kept[: note_end + 1]
+
     # Trim trailing heading-like short lines that don't belong to this table
     # (the start of the next section). Two patterns are trimmed:
     #   * Title-Case headings without a sentence terminator
@@ -1414,7 +1507,17 @@ def _extract_table_body_text(
         s = re.sub(r"[ \t]+", " ", ln).strip()
         if s:
             cleaned_lines.append(s)
-    return "\n".join(cleaned_lines).strip()
+    result = "\n".join(cleaned_lines).strip()
+    # Degenerate-prose guard (RC-T Layer-2, v2.4.97): drop a raw_text
+    # fallback that is really body prose the region overshot into, so the
+    # renderer emits a clean caption-only table instead of an
+    # ``unstructured-table`` dump that duplicates Results/Discussion prose.
+    # ``DOCPLUCK_RCT_L2_BYPASS`` reverts both Layer-2 additions (Note-anchor
+    # + this guard) to HEAD behavior — used only by the FP-scan harness to
+    # diff guard-live vs guard-bypassed over the full corpus.
+    if not os.environ.get("DOCPLUCK_RCT_L2_BYPASS") and _raw_text_is_degenerate_prose(result):
+        return ""
+    return result
 
 
 def _figure_from_caption(

{docpluck-2.4.96 → docpluck-2.4.98}/docpluck/tables/cell_cleaning.py

@@ -212,6 +212,12 @@ def _merge_continuation_rows(rows: list[list[str]]) -> list[list[str]]:
             return True
         if ")" in s and "(" not in s:  # close-paren tail: "8.34)", "10.28)"
             return True
+        # Close-bracket tail of a CI split across two rows: "[0.59," wraps and
+        # its upper bound "0.73]" lands on the next row (cog_emo Table 8 — the
+        # bracketed CI form, not the parenthetical one above). The parent cell
+        # ends with "," so the fragment joins after a space → "[0.59, 0.73]".
+        if "]" in s and "[" not in s:  # close-bracket tail: "0.73]", "−0.66]"
+            return True
         return False
 
     def _is_fragment_continuation(row: list[str], parent: list[str]) -> bool:
@@ -393,13 +399,32 @@ _NUMERIC_CELL_RE = re.compile(
     r"^[-−–]?\d+(?:[.,]\d+)*(?:[%∗*]+)?(?:\s*\([^)]*\))?$"
 )
 
+# A cell carrying a statistic VALUE (vs a header label). Broader than
+# _NUMERIC_CELL_RE: also matches APA leading-dot decimals (".34"), operator-
+# prefixed p-values ("< .001"), bracketed numeric intervals ("[0.53, 0.72]"),
+# and the "N/A" filler — all DATA, not header text. The interval branch requires
+# a digit and NO letters inside the brackets so a genuine header cell like
+# "[95% CI]" (letters present) is NOT counted as data and stays a header. Used by
+# `_is_header_like_row` so a real data row whose APA-formatted values the bare
+# numeric pattern under-counted is not mistaken for an extra header row — the
+# bug that silently dropped the FIRST data row of two-header-row correlation
+# tables (collabra.90203 Table 10, DP-5).
+_DATA_VALUE_CELL_RE = re.compile(
+    r"^(?:"
+    r"[<>=]?\s*[-−–]?\d*[.,]?\d+(?:[.,]\d+)*(?:[%∗*]+)?(?:\s*\([^)]*\))?"
+    r"|\[[^\]A-Za-z]*\d[^\]A-Za-z]*\]"
+    r"|n\s*/?\s*a"
+    r")$",
+    re.I,
+)
+
 
 def _is_header_like_row(row: list[str]) -> bool:
     """Heuristic: a row that looks like part of a header rather than data."""
     nonempty = [c.strip() for c in row if (c or "").strip()]
     if not nonempty:
         return False
-    numeric = sum(1 for c in nonempty if _NUMERIC_CELL_RE.match(c))
+    numeric = sum(1 for c in nonempty if _DATA_VALUE_CELL_RE.match(c))
     if numeric / len(nonempty) > 0.3:
         return False
     avg_len = sum(len(c) for c in nonempty) / len(nonempty)