docpluck 2.4.97__tar.gz → 2.4.98__tar.gz

{docpluck-2.4.97 → docpluck-2.4.98}/.claude/skills/_project/lessons.md

@@ -641,3 +641,15 @@ Rotation picks `pool[(N mod L) : (N mod L) + rotation_size]` wrapping. Over `cei
 **What (fix):** `api/cron/daily-digest` intermittently logged "Daily dispatch failed" (one-error admin digest email). Ground truth came from prod `system_logs.context.errorStack` (the logger persists `errorName`/`errorMessage`/`errorStack`; queried read-only via the Neon `DATABASE_URL` in `PDFextractor/frontend/.env.local`): two failed runs (06-17, 06-18) were transient **Neon serverless** errors — `terminating connection due to administrator command` (connection recycled on scale-to-zero) and a control-plane HTTP 500 tagged `"neon:retryable":true` — thrown at DIFFERENT queries, with adjacent days succeeding. NOT a logic bug: both unattended crons fire ~15 sequential neon-http calls at 03:00/09:00 UTC against likely-cold compute and the driver does no retries, so one blip aborts the whole run (and Vercel Cron does not re-run failed crons). Fix: new `frontend/src/lib/db-retry.ts` (`withDbRetry`/`isRetryableDbError` — 3 attempts, 250/500ms backoff, retries ONLY transient/connection-class signatures, never app errors) wrapping every DB call in `notifications/dispatch.ts` + `admin/blob-cleanup/route.ts` + the in-band `sendEmail` (safe to replay — `email_events` ON CONFLICT de-dupes). Regression test `frontend/src/lib/db-retry.test.ts` via `node --test` (Node 26 strips TS natively → zero new deps; first frontend unit test + `npm test` script; excluded from `next build` via tsconfig). Shipped commit `ea86e85`; post-deploy `?dryRun=1` smoke = HTTP 200 + clean DispatchResult.
 
 **How to detect next time:** (1) The admin digest aggregates the PREVIOUS 24h, so a "1 error" email naming `api/cron/daily-digest` is usually the prior day's transient blip, not a live outage — before theorizing, query `system_logs` for the real `errorName`/`errorStack` (it's `NeonDbError`, often `neon:retryable:true`). (2) ANY unattended Vercel cron doing a burst of neon-http queries needs `withDbRetry`: Neon scale-to-zero WILL recycle a connection mid-run and there is no driver-level retry. (3) `next build` here does NOT run eslint (Next 16 — see card `nextjs-16-build-does-not-run-eslint`), so a red `npm run lint` does not block deploy; keep the lint gate green separately for `/ship`.
+
+## 2026-06-25 — Branch can lag its own pushed tags; `check_app_pin_sync.py` PASS does NOT prove the branch is current
+
+**Symptom.** Session opened "clean", `check_app_pin_sync.py` PASS — yet `main` (local AND `origin/main`) was at the v2.4.95 commit while tags `v2.4.96`/`v2.4.97` were already pushed 4 commits ahead, the app pin had auto-bumped to `@v2.4.97`, and the working tree `__version__` read 2.4.95. So **production ran v2.4.97 while the library mainline + working tree claimed v2.4.95.** A prior session tagged+pushed without fast-forwarding the branch, and the tree was checked out backwards.
+
+**Why the gate didn't catch it.** `check_app_pin_sync.py` compares the app pin to the *latest tag* — both were v2.4.97, so PASS. It does not compare the *branch HEAD* to the latest tag. And its working-tree note had a direction bug: it told the BEHIND tree it was "ahead — tag+push v2.4.95" (would have tagged an OLDER version). Fixed the note (ordered `_vtuple()`, commit `59cff5b`); the branch-vs-tag check is still a manual step.
+
+**Rule.** At session start (and before any release), run `git tag --sort=-v:refname | head` and `git rev-parse main origin/main` and compare to the latest `v*` tag. If `main` is an ANCESTOR of the latest tag → the branch lags its own release; fast-forward (`git merge --ff-only v<latest>`) before iterating, or you'll work on a stale base and risk re-doing / colliding with shipped work. A green `check_app_pin_sync.py` is necessary but NOT sufficient evidence the branch is current. Companion: card `release-version-collision-with-parallel-uncommitted-stream`.
+
+## 2026-06-25 — Word-multiset preservation is BLIND to reading-order regressions; AI-verify is mandatory before any reorder flip
+
+The RC-1 banded column-correction flag passed a full-corpus word-preservation scan (26/26 baseline papers, 0 multiset violations) — looked safe to flip default-ON. The 8-canary AI-verify then found **3 ON_REGRESSION** the scan was structurally blind to: running-header furniture injected into prose, Abstract/Intro section order inverted, prose fragmented on a *single-column* paper (false-positive gutter). A pure reorder/furniture-injection preserves the word multiset exactly. **Never flip a reading-order-affecting default on word-preservation (or char-ratio / Jaccard) evidence alone — those gates cannot see "right words, wrong order / wrong place." AI-verify against the gold is required, and the bar is zero ON_REGRESSION across the full canary set.** Re-validates the project ground-truth hard rule.

{docpluck-2.4.97 → docpluck-2.4.98}/.claude/skills/docpluck-iterate/LEARNINGS.md

@@ -1327,3 +1327,44 @@ aren't skipped.
 ### Verification Gaps / Deferred (queued, not dropped)
 - **RC-T Layer-1 recovery** — actually RECOVERING lost table data via tight `table_areas` (plos_med T5's 13 SAE rows; chan_feldman T2 column-squish) — out of Option-A scope, deferred by the user.
 - **Audit of the ~37 `_strip_phantom` th-stripped tables** — some title-shaped `<th>` strips may be wrongly-stripped REAL tables (pre-existing, predates this cycle). Needs its own verification cycle (render each, judge real-vs-phantom). Surfaced explicitly; NOT silently shipped around.
+
+---
+
+## Run: 2026-06-25 · cycle 1 (investigation + verification) · @ v2.4.97 · verdict PARTIAL
+
+**Goal:** user invoked `/docpluck-iterate`; established stop-condition implicitly via the two scope decisions below. Targeted RC-T table-bbox (then, on evidence, the RC-1 banded flip).
+
+### What landed (clean, shipped, verified)
+1. **Branch reconciliation (the big one).** At session start `main` (local AND `origin/main`) was at the v2.4.95-era commit `2dbdd98`, but tags `v2.4.96`/`v2.4.97` were already pushed 4 commits AHEAD (real, test-backed RC-T + DP-2/DP-5 work) and the app pin had auto-bumped to `@v2.4.97` → **production ran v2.4.97 while the library mainline claimed v2.4.95.** Fast-forwarded `main` (local + origin) to `813aa4c` (v2.4.97); now mainline == latest tag == app pin. **Lesson: at session start, ALWAYS check `git tag` vs `git rev-parse main origin/main` — a prior session can push tags without advancing the branch, and the working tree can be checked out backwards. The `check_app_pin_sync.py` PASS hid it (pin==tag is true even when the *branch* lags the tag).**
+2. **`check_app_pin_sync.py` direction bug + test** (commit `59cff5b`). `compare()` said "working-tree X is ahead of tag Y — tag+push X" for ANY `X!=Y`; a tree BEHIND its tag (2.4.95 < 2.4.97, exactly this session) got told to tag an OLDER version. Fixed with ordered `_vtuple()`; `tests/test_check_app_pin_sync.py` (10 cases, fails-before/passes-after).
+
+### What was investigated and correctly NOT shipped
+3. **RC-T table-data recovery = the RC-1 banded layout-channel work (architectural).** ip_feldman T10 (and 3 other canary orphans) lose real table data; gold confirms a 7-row table, render drops all 7. Probed FOUR recovery angles (Camelot free-form, whitespace_cells full region, banded text-channel flag, band-clipped whitespace_cells) — each bottoms out in: full-page Camelot bbox / region-overshoot-into-prose / RC-1 interleave INSIDE the table region (table rows + Discussion prose share y-positions) / tight-kerned glued stat cells that defeat x-gap column detection (`pdfplumber_extract_words_unreliable`). The orphan `### Table N` heading is the DELIBERATE v2.4.55 clean-fail (keeps caption for `table_parity`), NOT the bug. **Lesson: reproduce + probe before trusting a "small fix" hypothesis — my B2-is-easy guess was disproven in ~15 min of probing, saving a mis-scoped cycle.** (card `reproduce-triage-defect-at-head`)
+4. **RC-1 banded default-flip ATTEMPTED, REJECTED by 8-canary AI-verify.** Word-preservation corpus scan = 26/26 PRESERVED, 0 violations (looked safe!). But the 8-canary AI-verify (8 parallel Sonnet, Claude Max) found **3 ON_REGRESSION** (deterministically confirmed): `ar_apa` flag-ON injects `M. Muraven / Journal` running-header furniture ×2 + inverts Abstract/Intro order; `chan_feldman` injects Power-Analysis prose into Extension; `maier` (SINGLE-COLUMN) fragments prose via a false-positive gutter. **THE LESSON (re-validates the project hard rule): the word-multiset gate is BLIND to reordering + furniture-injection — it passed 26/26 while the flag corrupted 3 canaries. AI-verify is necessary, not optional, before any reading-order default flip.** Flag stays default-OFF. Precise gap-5/6/7 blockers + a stricter zero-regression re-verify gate written to the RC-1 spec.
+
+### Verification-infra defect re-confirmed (surfaced, not fixed — needs portfolio decision)
+5. **The pre-push canary-audit.sh hook clobbered run-meta `phase_5d_runs` + `cycle_status` on BOTH pushes this session** — overwriting my honest FAIL verdicts with `AUDIT_DEFERRED_TO_AGENT → union PASS` placeholders (`cycle_status` flipped to PASS, a broken corpus reading green). Restored truthful verdicts after each push per memory `feedback_canary_audit_clobbers_phase5d`. **This bug actively manufactures false-green on a FAIL corpus; it bit twice in one session.** The fix (make the hook not write deferred-PASS, or write BLOCKED/AUDIT_DEFERRED instead of PASS) is a substrate change in `~/.claude/skills/_shared/iterate-loop/canary-audit.sh` — surfaced for a dedicated substrate cycle.
+
+### Open (queued, NOT dropped — rule 0e-bis)
+- 4 canary FAILs remain (ip_feldman T10, chan_feldman T6/T9, chandrashekar T2 table-data loss; plos_med sidebar interleave) → the RC-1 banded layout-channel + sidebar-interleave architectural cycle (spec `2026-06-08-rc1-region-aware-column-architecture.md`, sharpened this run).
+- Run verdict: **PARTIAL** — 2 clean fixes shipped; the canary corpus is NOT clean (honest punch-list above), so the standing corpus verdict is FAIL and the iterate-gate `--close` correctly cannot green-close.
+
+---
+
+## Run: 2026-06-25 (cont.) · RC-T char-level column recovery — FOUNDATION landed (v2.4.98), T10 wiring deferred
+
+### Outcome
+- **char_whitespace_cells (the char-level absolute-x-gap column detector) IMPLEMENTED + tested** in `docpluck/tables/whitespace.py`, wired as an automatic fallback inside `whitespace_cells` when the word path finds < 2 columns. This is the layout-channel piece that finally makes the "RC-T table-data recovery" tractable.
+
+### Blind spot closed (the key technical finding)
+- **Earlier this session I assessed T10 recovery as "architectural / multi-session, char detection too hard."** That was WRONG — disproven by probing char-level x-positions: ip_feldman Table 10's 3 stat columns are cleanly separable at the CHAR level (gaps 23-78pt) even though pdfplumber's WORD grouper glues the whole numeric row (`.29***−.21***.07`) into one token so the word-gap detector finds nothing. This is exactly the `pdfplumber_extract_words_unreliable` lesson ("always carry a char-level absolute-x-gap fallback") — I should have reached for it before declaring the wall.
+- **Right-aligned numeric tables need column-START-edge voting, not gap-MIDPOINT voting.** The label column is variable-width (`Loneliness` vs `Social orientation scale`), so gap midpoints scatter and never reach the stability threshold; but the DATA columns are left-aligned to fixed x, so their left edges are stable. `_find_stable_column_boundaries(bucket_pt>0)` votes on column starts + buckets within 8pt. With this, char_whitespace_cells recovers all 7 T10 data rows matching the gold exactly.
+
+### Edge case
+- **The word path MUST stay byte-identical when adding a char path.** First cut refactored `_find_stable_column_boundaries` to count only multi-column rows in the denominator — that silently changed the WORD path's stability threshold and broke 2 existing table tests (caught at ~65% of the broad run). Fix: branch the function — `bucket_pt<=0` runs the verbatim original word logic, `bucket_pt>0` runs the new char logic. Lesson: when extending a shared helper, keep the existing caller's path provably unchanged (literal copy), don't "improve" it in the same edit.
+
+### Deferred (queued, NOT dropped — rule 0e-bis)
+- **char_whitespace_cells alone does NOT close the T10 canary in production.** T10's caption is *matched* by the garbage full-page Camelot table, so it's added directly to `tables` and the whitespace fallback (where the char path lives) is never reached for it. Closing T10 needs the **degenerate-Camelot-table replacement** in `extract_structured.py`: when a Camelot-matched table has a full-page-ish bbox + furniture/prose cells, discard it and re-extract via `whitespace_cells` on the gutter-band-clipped caption region, PLUS a region prose-trim (the `(extension)` wrapped-caption + Discussion-prose tail still contaminate the region). That change touches the core extraction path for ALL papers → requires the full ~48-paper guard-live-vs-bypassed diff + 7-canary AI-verify before it ships (cycle-3 caption-follows revert is the precedent). Foundation is in place; this is the next focused cycle.
+
+### Verification
+- `tests/test_whitespace_char_fallback.py` (5 cases incl. real-PDF ip_feldman T10 = all 7 gold rows recovered; synthetic tight-kerned recovery; word-space reinsertion; delegation wiring; single-column-no-fabrication guard). 15/15 pass on the whitespace/char/caption-table suite; 0 failures across the broad table/render run; word path proven byte-identical (restored verbatim).

{docpluck-2.4.97 → docpluck-2.4.98}/CHANGELOG.md

@@ -1,5 +1,32 @@
 # Changelog
 
+## [2.4.98] — 2026-06-25
+
+**ESCIcheck handoff defects (DOCPLUCK_HANDOFF_2026-06-25): η²p effect-typing, inline correlation r-typing, and bracketed-CI continuation merge — flatten/cell-cleaning only, AI-gold-verified.** `TABLE_EXTRACTION_VERSION` → `2.4.4`. Render-visible in the inline flattened-table blocks + `.tables.jsonl` sidecar `fields`. **No capture-path change** — caption→table pairing is byte-identical to v2.4.3. Grounded against the AI `reading`/`stats` golds (article-finder), not pdftotext, and confirmed by a 6-paper AI-gold canary verify. Triage: `docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md`.
+
+- **DP-3 (table side) — type a font-dropped partial-η² effect column.** Many APA ANOVA tables report effect size as `η²p`, but the glyph lives in a font with no ToUnicode mapping (`pdffonts`: `uni:no`), so pdftotext AND pdfplumber both decode it as a space — the effect-size column header is blank and no "eta"/"η" token survives anywhere, so `tables.flatten._effect_key` fell back to the generic `est` key and the consumer got a nameless number. `flatten._infer_anova_eta2_hint` now types an unlabeled estimate column as `eta2` when the table is structurally an F-test/ANOVA results table (carries an `F` column + a `BF01` and/or CI column) and names NO competing effect (`d`/`dz`/`r`/`OR`/`g`); the value is range-guarded to η²'s domain `[0, 1]`. Keyed on the structural signature, never paper identity. `collabra.90203` Tables 8 & 9 now emit `eta2` matching the gold's `η²p` column (`.06, .00, .01, .04, …`). **The text-channel η²p (in body prose) is unrecoverable — OCR-tier won't-fix** (glyph identity absent from the PDF), as is DP-6 figure-inset text.
+
+- **DP-5 (b) — type inline self-labeled correlation cells.** A cell that states its own statistic — `r = .67`, `r = -.73`, `d = 0.32` — is now typed by that token even when the column header is generic ("Effect size") and the column-role recovery produced no estimate for it (`flatten._inline_stat_field`). `cog_emo` (`10.1080/02699931.2024.2434156`) Table 8's intercorrelation rows now carry a typed `r` with their CI/p, matching the gold. The display value is the number only, so the sentence renders `r = 0.63` (not `r = r = .67`).
+
+- **DP-5 (a) — merge a CI split across two rows by its bracket tail.** Camelot stream splits a wrapped CI cell — `[0.59,` on one row, `0.73]` on the next. `cell_cleaning._merge_continuation_rows` already folded the *parenthetical* tail (`8.34)`) but not the *bracketed* CI tail (`0.73]`); `_is_fragment_cell` now recognizes a close-bracket tail, so the bound rejoins (`[0.59, 0.73]`) and the bare-fragment junk row no longer survives (cog_emo Table 8: 14 → 10 rows, CIs complete).
+
+**Deferred (prototyped + REVERTED this cycle — surfaced, not dropped): DP-1 / DP-2 capture recovery.** A caption page-attribution fix (`find_caption_matches` advancing `char_start` past the leading `\f` its `^\s*` ate) does correct the off-by-one that mis-pages a table starting a page, and in isolation it unblocked `collabra.77859` Table 4's replication `t/df/d` (DP-1). But the **mandatory AI-gold canary verify showed it is net-harmful corpus-wide**: populating the previously-empty `line_text` re-scores `_find_caption_for_table`'s same-page token overlap and surfaces low-quality whitespace tables, mis-pairing tables whose captions share a page (efendic T4/T5, cog_emo T8/T9 swapped) and only half-fixing plos_med (Table 4 still wrong, Table 5 degraded). Reverted to keep the corpus regression-free. It needs same-page-caption disambiguation + whitespace-region quality gating, queued as its own gated cycle (full 48-paper guard-diff + canary AI-verify). DP-2's Table 1 (an 8-column grouped table) is beyond the lineless fallback regardless. See the triage doc.
+
+**Also fixed (pre-existing, leave-nothing-behind):** `test_table_html_renders_when_structured` and `test_table_kinds_are_valid` asserted a non-structured table must be `kind == "isolated"`, but the layout-channel `whitespace` fallback (a valid kind since the v2.4.3 foundation) can surface — including when Camelot returns "no tables" under cumulative test-suite load (memory `feedback_camelot_flake_cumulative_load`). The tests now accept `whitespace` (asserting cells + HTML present, no Camelot confidence), so a Camelot load-flake on the jama/ieee/chen fixtures no longer mis-fails them.
+
+Verification: shipped flatten/cell-cleaning changes covered by `tests/test_tables_flatten.py` (TestInlineSelfLabeledStat ×3, TestBracketTailCIContinuation) + `tests/test_tables_flatten_blank_header_recovery.py::TestT90203Table8BlankHeaders` (eta2 typed, est gone). Full library suite green under `pytest -n10` (real-PDF Camelot tests flake only under serial cumulative load — each passes per-file / under xdist). 6-paper AI-gold canary verify confirms DP-3 (collabra.90203 η²p), DP-5 (cog_emo r + CI) correct; it is what caught the DP-1/DP-2 page-fix regression and drove its revert.
+
+---
+
+**RC-T foundation (earlier same-day commit): char-level column-detection fallback for tight-kerned tables.** `TABLE_EXTRACTION_VERSION` → `2.4.3` (now → `2.4.4` per above). Library-internal capability addition — a foundation for the still-open RC-T Layer-1 table-data recovery.
+
+- **`docpluck/tables/whitespace.py` gains `char_whitespace_cells`** — a char-level absolute-x-gap column detector, wired as an automatic fallback inside `whitespace_cells` that fires ONLY when the word-gap detector finds < 2 columns. On tight-kerned PDFs pdfplumber's word grouper glues a whole numeric row into one token (e.g. ip_feldman Table 10's `.29***−.21***.07`), so the word path finds no column gaps; the chars are still cleanly separated by large inter-column gaps. The fallback recovers the grid from char x-gaps, **voting on column-START edges** (in a right-aligned numeric table the label column is variable-width so gap midpoints scatter, but the data columns are left-aligned to fixed x), with an 8pt boundary bucket, and reinserts intra-cell word spacing from glyph geometry.
+- **Strictly additive — currently-correct tables are byte-unchanged.** The word path (`_find_stable_column_boundaries(bucket_pt=0)`) is restored verbatim to its pre-change behavior; only a region that previously yielded NO whitespace grid (`whitespace_cells` returned `[]`) can now gain one, and only when char-columns clear a 60%-row stability bar. In isolation this recovers all 7 of ip_feldman Table 10's regression-coefficient rows matching the AI gold.
+
+**Not yet closed (next gated cycle):** the char fallback alone does NOT fix Table 10 in the live pipeline — T10's caption is *matched* by a degenerate full-page Camelot table, which is used directly, so the whitespace fallback (where the char path lives) is never reached. Closing it needs an `extract_structured.py` change — detect a degenerate matched Camelot table (full-page-ish bbox + furniture/prose cells), discard it, and re-extract via `whitespace_cells` on the gutter-band-clipped caption region, plus a region prose-trim — which touches the core extraction path for all papers and so requires the full ~48-paper guard-diff + 7-canary AI-verify before it ships. See `LEARNINGS.md` 2026-06-25 and `docs/superpowers/specs/2026-06-21-rc-t-table-region-prose-contamination.md`.
+
+Verification: `tests/test_whitespace_char_fallback.py` (5 cases incl. real-PDF ip_feldman Table 10 = all 7 gold rows recovered, synthetic tight-kerned recovery, intra-cell word-space reinsertion, delegation wiring, single-column-no-fabrication guard); 15/15 pass on the whitespace/char/caption-table suite across two independent runs; 0 failures across the broad table/render suite; word path proven byte-identical.
+
 ## [2.4.97] — 2026-06-22
 
 **Three table fixes shipped together (combined from two concurrent sessions): type the skipped p+df columns (DP-2), stop dropping / mis-binding two-header-row tables (DP-5), and stop the table raw_text fallback swallowing body prose (RC-T Layer-2).** `TABLE_EXTRACTION_VERSION` → `2.4.2`; no `NORMALIZATION_VERSION` / `SECTIONING_VERSION` change. DP-2/DP-5 are render-visible in the inline flattened-table blocks + the `.tables.jsonl` sidecar `fields` (the `<table>` HTML gains the previously-dropped data rows); RC-T Layer-2 is render-visible in the `unstructured-table` fallback blocks. DP-2/DP-5 filed in `ESCIcheckapp/docs/DOCPLUCK_HANDOFF_2026-06-21.md`; RC-T Layer-2 per `docs/superpowers/specs/2026-06-21-rc-t-table-region-prose-contamination.md`.

{docpluck-2.4.97 → docpluck-2.4.98}/LESSONS.md

@@ -274,6 +274,74 @@ Cite: `docpluck/tables/camelot_extract.py::_augment_lattice_with_stream_rows` +
 
 ---
 
+## L-010 — A caption that starts a page is mis-paged by the `^\s*` form-feed; and a font with no ToUnicode makes its glyph unrecoverable (recover the column ROLE, not the glyph)
+
+### Two findings, both surfaced 2026-06-25 by the ESCIcheck handoff
+
+1. **The `^\s*` caption regex eats the `\f` and mis-pages a page-starting table.**
+   `TABLE_CAPTION_RE` / `FIGURE_CAPTION_RE` begin `^\s*`. When a table starts a new
+   page, pdftotext emits `…results\n\fTable 4. …`, and `\s*` consumes the `\f`, so
+   `m.start()` lands *before* the form-feed. `_page_for_offset` then counts the
+   caption on the page BEFORE the break (off-by-one), and `_line_at` returns the
+   empty pre-`\f` segment (so `line_text == ''`). With the wrong page,
+   `_bbox_of_caption_line` can't find the caption in the layout channel → the whole
+   layout-region lookup returns None → the whitespace/char fallback never fires and
+   the table degrades to a caption-only stub. On `collabra.77859` this hit **all 5
+   tables**. The seemingly-obvious fix — advance `char_start` past the leading `\f`
+   to the actual "Table"/"Figure" token (`captions.find_caption_matches`) — DOES
+   correct the page AND unblock `collabra.77859` Table 4's replication stats (DP-1) in
+   isolation. **It was tried 2026-06-25 and REVERTED.** Populating the
+   previously-empty `line_text` re-scores `_find_caption_for_table`'s same-page
+   token-overlap and surfaces low-quality whitespace tables, so the mandatory AI-gold
+   canary verify caught it mis-pairing tables whose captions share a page (efendic
+   T4/T5, cog_emo T8/T9 swapped) and only half-fixing plos_med. **The lesson is the
+   process, not the patch:** a capture-path change that helps one paper in isolation
+   can silently mis-pair others — only a corpus-wide AI-gold verify (NOT the unit
+   suite, which stayed green) reveals it. The real fix needs same-page-caption
+   disambiguation in `_find_caption_for_table` + whitespace-region quality gating
+   FIRST; queued as its own gated cycle. Symptom to watch: `line_text == ''` on a
+   caption, or a caption whose `page` is one less than where the table visibly is.
+
+2. **A glyph with no ToUnicode mapping is gone from BOTH channels — recover its
+   ROLE, not the glyph.** `collabra.90203` reports `η²p`, but `pdffonts` shows the
+   symbol font as `uni: no` (no ToUnicode CMap), so pdftotext AND pdfplumber both
+   decode the glyph as U+0020 — the text reads `( = .000, …)` and the table's
+   effect-column header is blank. This is the same class as the residual deleted-minus
+   (memory `project_docpluck_rc_b7_done_w0h`): the *character identity* is absent from
+   the PDF, so text-channel recovery is **OCR-tier won't-fix**. But in a TABLE the
+   column's *role* is recoverable from structure — an F-test/ANOVA results table that
+   reports a Bayes factor + CI and names no competing effect reports η²p by APA
+   convention — so type the value `eta2` from the structural signature (range-guarded
+   to η²'s `[0,1]` domain) even though the glyph is unrecoverable
+   (`flatten._infer_anova_eta2_hint`). Don't chase the glyph; recover the meaning.
+
+### The rules
+
+1. **A caption-page / capture-path change MUST be AI-gold-verified across the corpus
+   before shipping — the unit suite will not catch a mis-pairing.** The page-fix kept
+   all 1852 unit tests green yet swapped table↔caption pairings on 3 papers. Per the
+   project ground-truth rule, render the canary set and compare TABLES against the AI
+   `reading` golds; revert if any paper regresses. (`^\s*`-anchored scans that skip
+   `\f` ARE the right idea for page attribution, but the downstream `line_text` /
+   same-page-caption scoring must be made robust in the same change, not after.)
+2. **Before "the symbol got stripped", run `pdffonts`.** If the glyph's font is
+   `uni:no`, nothing in the byte stream carries its identity — stop trying to recover
+   the glyph; recover the column's role/meaning from structure, or mark it OCR-tier.
+   (Shipped: `flatten._infer_anova_eta2_hint` types the value `eta2` from the F-test
+   table structure even though the η²p glyph is gone.)
+3. **A self-labeled cell beats its column header.** `r = .67` / `d = 0.32` states its
+   own type; type by the cell token even under a generic "Effect size" header
+   (`flatten._inline_stat_field`, shipped) — store only the numeric part so the
+   sentence assembler doesn't double the prefix (`r = r = .67`).
+
+Cite (SHIPPED v2.4.4): `docpluck/tables/flatten.py::_infer_anova_eta2_hint` +
+`_inline_stat_field`, `docpluck/tables/cell_cleaning.py::_is_fragment_cell` (bracket-CI
+tail). REVERTED (queued): the `captions.find_caption_matches` char_start advance +
+`whitespace._whitespace_grid_is_clean` / `_trim_trailing_prose_rows` gates.
+See `docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md`, CHANGELOG v2.4.98.
+
+---
+
 ## When to add a new lesson here
 
 Add a lesson when:

{docpluck-2.4.97 → docpluck-2.4.98}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.97
+Version: 2.4.98
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://docpluck.app
 Project-URL: Documentation, https://docpluck.app/api-docs

{docpluck-2.4.97 → docpluck-2.4.98}/docpluck/__init__.py

@@ -78,7 +78,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.97"
+__version__ = "2.4.98"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.97 → docpluck-2.4.98}/docpluck/extract_structured.py

@@ -37,7 +37,7 @@ from .tables.render import cells_to_html
 from .telemetry import record_fallback
 
 
-TABLE_EXTRACTION_VERSION = "2.4.2"  # v2.4.2 (RC-T Layer-2): _extract_table_body_text now (a) Note-anchor — a table's "Note:" footnote is its last element, so trim body prose bled past it (chan_feldman T1/T3, efendic_2022 T5); and (b) degenerate-prose guard — suppress a raw_text fallback that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so render emits a clean caption-only table instead of an unstructured-table dump duplicating Results/Discussion prose (chan_feldman T9 was a verbatim ## Discussion duplicate). FP-safe (real cells start with header/label/number/single-letter marker, never a wrapped continuation); full-corpus 101-PDF guard-diff only trims+suppresses (grew=0 changed=0). # v2.4.1 (DP-2/DP-5): (DP-2) blank-header role recovery now types the unlabeled p-value (a bare `.XXX` after the test stat, no comparison op) and df (a bare integer/Welch-decimal between the stat and the d[CI] column) columns it previously skipped — collabra.77859 T3 fields gain p+df (tables.flatten._recover_blank_roles Pass 4.5). (DP-5) parallel-arm tables with a TWO-ROW header no longer drop their first data row, and a CENTERED super-header is aligned to its arm block instead of its visual-center column: (a) cell_cleaning._is_header_like_row counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, N/A) as data via _DATA_VALUE_CELL_RE so a real first data row isn't read as a 3rd header row (collabra.90203 T10 recovered the Identifiable/Explicit-learning correlation); (b) tables.flatten._detect_column_groups re-derives arm boundaries from equal-width blocks of the data region (each must hold one super-label) so a centered super-label folded mid-span no longer swaps arm values (xiao_2021 T4 Original/Replication F) or pushes a stat column into the label region; (c) tables.flatten._classify_column reads a folded super-header cell's role from its sub-part (collabra.90203 T10 CI). Full-corpus cached-table flatten diff: no clean-table regression. # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
+TABLE_EXTRACTION_VERSION = "2.4.4"  # v2.4.4 (ESCIcheck handoff 2026-06-25 — flatten-only, AI-gold-verified): (DP-3) flatten._infer_anova_eta2_hint types a font-dropped η²p effect column by STRUCTURE — an unlabeled estimate column in an F-test/ANOVA results table (F + BF01/CI, no competing d/r/OR) is keyed `eta2`, range-guarded to [0,1]; the η²p glyph itself is unrecoverable (NotoSerif uni:no, OCR-tier). (DP-5b) flatten._inline_stat_field types a self-labeled cell (`r = .67`, `d = 0.32`) by its token even under a generic "Effect size" header. (DP-5a) cell_cleaning._is_fragment_cell recognizes a bracketed-CI close tail (`0.73]`) so a CI split across rows rejoins (`[0.59, 0.73]`) and the junk fragment row is dropped (cog_emo T8 14→10 rows). These three are flatten/cell-cleaning ONLY — no capture-path change, so caption→table pairing is byte-identical to v2.4.3. (DP-1/DP-2 capture recovery — the caption page-attribution fix — was prototyped but REVERTED: AI-gold canary verify showed it mis-pairs same-page-caption tables (efendic T4/T5, cog_emo T8/T9) and only half-fixes plos_med; it is queued as its own gated cycle with same-page disambiguation + region-quality gating. See docs/TRIAGE_2026-06-25_escicheck_handoff_defects.md.) # v2.4.3 (RC-T foundation): tables.whitespace gains a CHAR-LEVEL column-detection fallback (char_whitespace_cells) — when pdfplumber's word grouper glues a tight-kerned numeric row into one token so the word-gap detector finds < 2 columns, recover the grid from char x-gaps, voting on column-START edges (right-aligned data columns are left-edge-stable even when the label column is variable-width) and reinserting intra-cell word spacing from geometry. Fires ONLY as a fallback when the word path returns < 2 columns, so currently-correct tables are byte-unchanged (word path restored verbatim). Recovers ip_feldman Table 10's 7 regression rows in isolation. NOT yet wired to replace a degenerate MATCHED Camelot table (that extract_structured change + region prose-trim is the next gated cycle — see LEARNINGS 2026-06-25). # v2.4.2 (RC-T Layer-2): _extract_table_body_text now (a) Note-anchor — a table's "Note:" footnote is its last element, so trim body prose bled past it (chan_feldman T1/T3, efendic_2022 T5); and (b) degenerate-prose guard — suppress a raw_text fallback that STARTS mid-sentence with a lowercase multi-letter word AND is majority sentence-shaped prose, so render emits a clean caption-only table instead of an unstructured-table dump duplicating Results/Discussion prose (chan_feldman T9 was a verbatim ## Discussion duplicate). FP-safe (real cells start with header/label/number/single-letter marker, never a wrapped continuation); full-corpus 101-PDF guard-diff only trims+suppresses (grew=0 changed=0). # v2.4.1 (DP-2/DP-5): (DP-2) blank-header role recovery now types the unlabeled p-value (a bare `.XXX` after the test stat, no comparison op) and df (a bare integer/Welch-decimal between the stat and the d[CI] column) columns it previously skipped — collabra.77859 T3 fields gain p+df (tables.flatten._recover_blank_roles Pass 4.5). (DP-5) parallel-arm tables with a TWO-ROW header no longer drop their first data row, and a CENTERED super-header is aligned to its arm block instead of its visual-center column: (a) cell_cleaning._is_header_like_row counts APA value shapes (leading-dot decimal, bracketed CI, operator-prefixed p, N/A) as data via _DATA_VALUE_CELL_RE so a real first data row isn't read as a 3rd header row (collabra.90203 T10 recovered the Identifiable/Explicit-learning correlation); (b) tables.flatten._detect_column_groups re-derives arm boundaries from equal-width blocks of the data region (each must hold one super-label) so a centered super-label folded mid-span no longer swaps arm values (xiao_2021 T4 Original/Replication F) or pushes a stat column into the label region; (c) tables.flatten._classify_column reads a folded super-header cell's role from its sub-part (collabra.90203 T10 CI). Full-corpus cached-table flatten diff: no clean-table regression. # v2.4.0 (REQUEST_11): flatten now populates fields for NON-clinical result tables — (a) blank-header column-role recovery (tables.flatten._recover_blank_roles): assign a stat role to a header-stripped column from its data-token SHAPE (CI brackets, df1/df2 pair, estimate-adjacent-CI, p-with-operator) AND caption/footnote/all-header-rows vocabulary, never bare position; recovers collabra.77859 T5 (t/df/d/CI) + collabra.90203 T8/T9 (F/df/p/BF01/eta²p-as-est/CI). (b) packed parallel-arm split (tables.flatten._detect_packed_arms/_flatten_packed_arms): tables packing k≥2 arms into single cells ("Separate Joint" + space-joined values) emit one typed record per arm (group=arm) — collabra.77859 T3 Separate/Joint, xiao_2021 T7 Regret/Justifiability. (c) new BF01 role; validity guards drop r∉[-1,1] / non-monotone CI / non-int n / p∉[0,1]. (d) GENERAL L-004 fixes: _parse_number + _parse_ci_cell fold U+2212 MINUS (negative t/d/CI bounds in Camelot cells were dropped/sign-lost); _VALUE_GROUP_RE handles bracket-led CI groups. Default render + PROSECCO output byte-identical. # v2.3.0 (Tier-2, REQUEST_10): cross-flavor lattice-augmentation — recover data rows a lattice extraction vertically TRUNCATED by appending the rows a same-page, same-column-count stream table captured below the lattice bbox (camelot_extract._augment_lattice_with_stream_rows), gated on equal-col-count + bbox overlap + extends-below; PLUS numeric/parenthetical continuation merge (cell_cleaning._merge_continuation_rows) rejoining stream's stacked value/parenthetical cells. Fixes PROSECCO Table 2 R2-R6. v2.2.0: EC-T1 docpluck.tables.flatten — per-row FlattenedRow records (sentence + structured fields) for downstream stat-verification consumers (effectcheck/escimate/scimeto) + opt-in inline "rendered as text" block below each <table> via render_pdf_to_markdown(flatten_tables_inline=True). v2.1.5: cell-cleaning recovers CMEX10 extensible-bracket PUA glyphs (U+F8EE-F8FB). v2.1.4: cell-cleaning recovers Adobe-Symbol-font PUA glyphs (beta/chi/bullet as U+F0xx). v2.1.3: cell-cleaning recovers '<'-as-backslash glyph corruption. v2.1.2: cell-cleaning recovers descending-CI '2'-for-minus corruption. v2.1.1: cell-cleaning recovers (cid:0) corrupted minus signs + strips math-alphanumeric styling. v2.1.0: cell-cleaning pipeline ported from splice spike (multi-row header detection, continuation merging, leader-dot strip, mash-split, group separators, sig-marker attach)
 
 TableTextMode = Literal["raw", "placeholder"]
 

{docpluck-2.4.97 → docpluck-2.4.98}/docpluck/tables/cell_cleaning.py

@@ -212,6 +212,12 @@ def _merge_continuation_rows(rows: list[list[str]]) -> list[list[str]]:
             return True
         if ")" in s and "(" not in s:  # close-paren tail: "8.34)", "10.28)"
             return True
+        # Close-bracket tail of a CI split across two rows: "[0.59," wraps and
+        # its upper bound "0.73]" lands on the next row (cog_emo Table 8 — the
+        # bracketed CI form, not the parenthetical one above). The parent cell
+        # ends with "," so the fragment joins after a space → "[0.59, 0.73]".
+        if "]" in s and "[" not in s:  # close-bracket tail: "0.73]", "−0.66]"
+            return True
         return False
 
     def _is_fragment_continuation(row: list[str], parent: list[str]) -> bool:

{docpluck-2.4.97 → docpluck-2.4.98}/docpluck/tables/flatten.py

@@ -252,6 +252,96 @@ def _effect_key(header_cell: str, hint: Optional[str]) -> str:
     return _EFFECT_TYPE_KEY.get(et or "", "est")
 
 
+# Header tokens that name a *competing* effect size — if any of these is present
+# in a table, an unlabeled estimate column is NOT safe to assume is partial-η².
+_COMPETING_EFFECT_RE = re.compile(
+    r"\bd\s*z?\b|\bd\s+or\s+d\b|cohen|hedges|\bg\b|"
+    r"(?<![A-Za-z])r(?![A-Za-z])|\bOR\b|odds\s*ratio|risk\s*diff|\bMD\b",
+    re.I,
+)
+
+# A cell that SELF-LABELS its statistic inline, e.g. "r = .67", "r = -.73",
+# "d = 0.32", "dz = 0.40", "η² = .008". The token names the field; the number is
+# the value. This is the strongest possible signal (the cell states its own
+# type), so it types a value even when the COLUMN header is generic ("Effect
+# size") or unrecognized — ESCIcheck DP-2026-06-25-5, cog_emo Table 8, where the
+# correlation column is headed "Effect size" not "r". Group 1 = token, group 2 =
+# signed value (APA leading-dot allowed). Anchored at cell start.
+_INLINE_STAT_CELL_RE = re.compile(
+    r"^\s*(r|d\s*z|dz|d|g|η\s*²?\s*p?|η2p?|eta\s*p?2?)\s*=\s*"
+    r"([-+]?\d*\.?\d+)\s*$",
+    re.I,
+)
+# Inline token → canonical field key. (Mirrors `_EFFECT_TYPE_KEY`, but keyed on
+# the raw token a cell uses rather than an `_effect_type_for` name.)
+_INLINE_STAT_KEY: list[tuple[re.Pattern[str], str]] = [
+    (re.compile(r"^η|^eta", re.I), "eta2"),
+    (re.compile(r"^d\s*z$|^dz$|^d$", re.I), "d"),
+    (re.compile(r"^g$", re.I), "d"),  # Hedges g shares the d-family field
+    (re.compile(r"^r$", re.I), "r"),
+]
+
+
+def _inline_stat_field(cell: str) -> Optional[tuple[str, float]]:
+    """If a cell self-labels its statistic (``"r = .67"``), return
+    ``(field_key, value)``; else ``None``. Value-domain guards (r∈[-1,1],
+    η²∈[0,1]) are applied by the caller's universal validity pass."""
+    m = _INLINE_STAT_CELL_RE.match((cell or "").replace("−", "-"))
+    if not m:
+        return None
+    tok = m.group(1)
+    try:
+        val = float(m.group(2))
+    except ValueError:
+        return None
+    for pat, key in _INLINE_STAT_KEY:
+        if pat.match(tok.replace(" ", "")):
+            return key, val
+    return None
+
+
+def _infer_anova_eta2_hint(
+    roles: dict[int, str],
+    header: list[str],
+    explicit_hint: Optional[str],
+) -> bool:
+    """Structural signal that an UNLABELED estimate column in this row is a
+    partial eta-squared (η²p).
+
+    Why this exists: on PDFs whose η²p header glyph has no ToUnicode mapping
+    (e.g. NotoSerif-Regular `uni:no`), pdftotext AND pdfplumber both decode the
+    glyph as a space, so the effect-size column header is blank *and* the table
+    vocabulary contains no "eta"/"η" token — `_effect_type_for` returns None and
+    the value falls back to the generic `est` key, leaving the consumer with a
+    nameless number (ESCIcheck DP-2026-06-25-3). The glyph itself is
+    unrecoverable, but the column's *role* is recoverable from structure: an
+    F-test / ANOVA results table that reports a Bayes factor and/or a CI and
+    names no competing effect size reports its effect as η²p by APA convention.
+
+    Fires ONLY when (keyed on a structural signature, never paper identity):
+      * no explicit effect type was named anywhere (header text + table vocab),
+      * the row carries an ``F`` statistic (it is an F-test / ANOVA row),
+      * the row carries a ``BF01`` and/or a CI column (the results-table shape),
+      * NO column header names a competing effect (d / dz / r / OR / g / …).
+
+    The caller additionally range-guards the value to η²'s domain [0, 1] before
+    applying the inferred key, so a stray non-proportion estimate is never
+    mistyped. Returns True ⇒ treat unlabeled `est` columns as η²p.
+    """
+    if explicit_hint:
+        return False
+    present = set(roles.values())
+    if "F" not in present:
+        return False
+    if not ({"BF01", "CI", "est_ci", "CI_lo", "CI_hi"} & present):
+        return False
+    if {"d", "r"} & present:
+        return False
+    if any(_COMPETING_EFFECT_RE.search(h or "") for h in header):
+        return False
+    return True
+
+
 def _classify_column(header: str) -> Optional[str]:
     """Return the canonical role for a column header, or None if unrecognized.
 
@@ -960,6 +1050,14 @@ def _flatten_one_row(
         elif role is not None:
             roles[ci] = role
 
+    # Structural η²p inference for an unlabeled estimate column in an F-test
+    # results table whose effect-size header glyph was dropped by the font
+    # (see `_infer_anova_eta2_hint`). Computed once per row from the resolved
+    # role set; only used below to type an `est`/`est_ci` column when no explicit
+    # effect was named, and only after the value passes η²'s [0, 1] range guard.
+    eta2_inferred = _infer_anova_eta2_hint(roles, header, effect_hint)
+    effect_hint_eff = effect_hint or ("eta_squared_partial" if eta2_inferred else None)
+
     # Pull per-role values out of the body row.
     role_vals: dict[str, str] = {}
     role_nums: dict[str, float] = {}
@@ -993,7 +1091,17 @@ def _flatten_one_row(
             if est is None:
                 est = _to_signed_float(raw)
             if est is not None:
-                eff_key = _effect_key(header[ci], effect_hint)
+                eff_key = _effect_key(header[ci], effect_hint_eff)
+                # When the η²p key was INFERRED (not named in the header), guard
+                # it to η²'s domain [0, 1] — a non-proportion estimate falls back
+                # to the generic `est` so a stray value is never mistyped as η²p.
+                if (
+                    eff_key == "eta2"
+                    and not _effect_type_for(header[ci])
+                    and eta2_inferred
+                    and not (0.0 <= est <= 1.0)
+                ):
+                    eff_key = "est"
                 role_nums[eff_key] = est
                 role_vals[eff_key] = raw
         elif role == "est_ci":
@@ -1001,7 +1109,14 @@ def _flatten_one_row(
             # or "0.76 [.50, 1.02]".
             est = _parse_leading_number(raw)
             if est is not None:
-                eff_key = _effect_key(header[ci], effect_hint)
+                eff_key = _effect_key(header[ci], effect_hint_eff)
+                if (
+                    eff_key == "eta2"
+                    and not _effect_type_for(header[ci])
+                    and eta2_inferred
+                    and not (0.0 <= est <= 1.0)
+                ):
+                    eff_key = "est"
                 role_nums[eff_key] = est
                 role_vals[eff_key] = (
                     re.split(r"[\[(]", raw, maxsplit=1)[0].strip() or _fmt_num(est)
@@ -1018,6 +1133,26 @@ def _flatten_one_row(
             if n is not None:
                 role_nums[role] = n
 
+    # Inline self-labeled statistic recovery: a cell like "r = .67" / "d = 0.32"
+    # states its own type, so type it even when the COLUMN header is generic
+    # ("Effect size") and the role loop above produced no estimate for it
+    # (ESCIcheck DP-2026-06-25-5). Scans every non-label cell; a role-loop value
+    # already present for that field wins (we never overwrite a typed value).
+    for ci in range(len(row)):
+        if ci == label_col:
+            continue
+        hit = _inline_stat_field(row[ci])
+        if hit is None:
+            continue
+        key, val = hit
+        if key in role_nums:
+            continue
+        role_nums[key] = val
+        # Store only the numeric part as the display value — the sentence
+        # assembler re-prefixes the field name ("r = …"), so keeping the inline
+        # "r = " here would double it ("r = r = .67").
+        role_vals[key] = _fmt_num(val)
+
     # Consolidate separate CI_lo / CI_hi columns into CI_lower / CI_upper so
     # the assembler sees a single CI primitive.
     if "CI_lo" in role_nums and "CI_hi" in role_nums: