docpluck 2.4.76__tar.gz → 2.4.80__tar.gz

{docpluck-2.4.76 → docpluck-2.4.80}/.claude/skills/_project/canary.json

@@ -56,6 +56,12 @@
     "locator_via_article_finder": "python ~/.claude/skills/article-finder/cache-check.py <doi>",
     "gold_via_article_finder": "python ~/.claude/skills/article-finder/ai-gold.py get <key> --view reading",
     "render_command": "python tools/render_for_audit.py --key {key} --out {out}",
+    "allowed_omissions": [
+      "Front-matter masthead block between the title (H1) and the Abstract: author name line(s), affiliation line(s), corresponding-author contact line, journal-name banner, volume/issue/page range, 'Article type', publisher/copyright line, 'Article reuse guidelines', journal-home URL. docpluck strips this masthead by design (render.py _strip_frontmatter_masthead_block); consumers needing structured author metadata use CrossRef/DOI lookup. Its absence from the rendered .md is NOT TEXT-LOSS or METADATA-LEAK.",
+      "Publication-history date lines such as 'Received <date>; revision accepted <date>' or 'Received/Revised/Accepted' mastheads — stripped as journal furniture (normalize.py _PAGE_FOOTER_LINE_PATTERNS).",
+      "Running headers / running feet (e.g. 'Ip and Feldman', 'PERSONALITY AND SOCIAL PSYCHOLOGY BULLETIN'), bare page numbers, standalone DOI/ORCID banner lines, and copyright lines — stripped as page furniture.",
+      "ORCID iD lines and 'Data Availability'/'Open Practices' badge furniture that pdftotext renders as standalone fragments."
+    ],
     "audit_subagent_prompt": "~/.claude/skills/_shared/iterate-loop/audit-subagent-prompt.md",
     "audit_subagent_model": "sonnet",
     "audit_invocation": {

{docpluck-2.4.76 → docpluck-2.4.80}/.claude/skills/_project/lessons.md

@@ -1,4 +1,34 @@
 
+## Stripping load-bearing front-matter metadata exposes pre-existing wrapped-title duplicates (2026-05-26 Cluster E revert)
+
+**What:** Cycle 4 of run 11 added line patterns to strip bare article ID (`1327169`) + article-type code (`research-article2025`) at top of PSPB-layout docs. Patterns smoke-tested clean (zero false positives across 20 cases). Render showed top-of-doc metadata correctly gone — but introduced `### Title duplicate` as wrapped multi-line text immediately under the H1. Root cause: pdftotext emits the title TWICE on PSPB layouts (main + running-header copy in column 2). The metadata lines were absorbing/separating the duplicate so it never reached `_promote_isolated_titlecase_subsection_headings`. Without them, the wrap candidate is now isolated and gets promoted.
+
+**How to detect:** any metadata-strip cycle where the BEFORE render had multi-line text just under the H1 (before the author byline) needs a wrapped-title-duplicate check AFTER the strips. Compare H1 token set vs the next 5-10 non-blank lines' token sets — if there's high overlap, the lines under the H1 are likely a duplicate that the metadata block was hiding.
+
+**Fix:** strip metadata + install a wrapped-title-duplicate detector in the same change. Don't ship the strip alone. The duplicate detector should match a paragraph-block under the H1 whose concatenated text equals the H1 modulo whitespace (or whose tokens are a high-overlap subset). Run AFTER the strips, BEFORE `_promote_isolated_titlecase_subsection_headings` so the wrap doesn't get promoted to `### `.
+
+**File:** `docpluck/normalize.py::_FRONTMATTER_LEAK_LINE_PATTERNS` had the `_ARTICLE_TYPE_CODE` + `_BARE_ARTICLE_ID` additions reverted; the safer `Article reuse guidelines:` leaf-node P0 pattern was kept (it's not load-bearing).
+
+## Subsection-chain promotion needs (a) parent-section blacklist AND (b) strict-adjacent backward walk (2026-05-26 Cluster A-ter)
+
+**What:** Stacked Method subsections (e.g., `## Method` immediately followed by `Design and Procedure` + blank + `Power Analysis and Sensitivity Test` + blank + body) were not being promoted to `### ` headings because the existing `_promote_isolated_titlecase_subsection_headings` cell-region reject + sibling-label reject correctly reject each candidate individually but can't see across the chain to confirm "this is a real stacked subsection set." The chain detection helper (`_is_subsection_chain_member`) closes that gap.
+
+**Two safety guards are mandatory:**
+1. **`_CHAIN_REJECT_PARENTS` blacklist** — when the chain's parent is `## Author Contributions` / `## CRediT` / `## Funding` / `## Acknowledgments` / etc., the candidates underneath are list items (CRediT roles, ORCID names, funding agencies), NOT subsection headings. Walking back to find the parent label and rejecting these is essential — otherwise chan_feldman's "Methodology" CRediT role gets promoted to `### Methodology` (the existing `test_chan_feldman_no_credit_role_methodology_heading` test catches this regression).
+2. **Strict-adjacent backward walk** (don't traverse through body) — a through-body backward walk over-promotes Table 4 row labels on ip_feldman ("Exploratory open-ended" / "Well-being measures and traits" / "IV1: estimation of negative emotional events" — these look like chain members under `## Method` if you walk through body). Strict-adjacent (only blank-separated candidates count) avoids this trap.
+
+**How to detect:** after any chain-promotion change, render and grep both: (a) `^### Methodology` on chan_feldman (must be 0) and (b) `^### Exploratory open-ended` / `^### IV1:` on ip_feldman (must be 0).
+
+**File:** `docpluck/render.py::_is_subsection_chain_member` (helper) + `_CHAIN_REJECT_PARENTS` frozenset (blacklist) + integration in `_promote_isolated_titlecase_subsection_headings` (bypass cell-region + sibling-label rejects when chain confirmed).
+
+## Orphan affiliation wrap-tail needs a tight line-level pattern with 60-char length lookahead (2026-05-26 Cluster C-bis)
+
+**What:** Cluster C's name-led-affiliation pattern in `_FRONTMATTER_LEAK_PARA_PATTERNS` matches the first line of a 2-line wrapped corresponding-author paragraph (`"Gilad Feldman, Department of Psychology, University of Hong Kong, Pok"`), but the wrap-tail (`"Fu Lam, Hong Kong SAR."`) survives because line-by-line iteration in `_strip_frontmatter_metadata_leaks` can't see across the boundary. The 2026-05-25 Cluster C run cleared finding #1 mostly but left this orphan.
+
+**Fix shape:** `^(?=.{1,60}$) <1-3 title-case place tokens>, <region: title-case+all-caps OR all-caps+optional-zip OR title-case>\.\s*$`. The 60-char lookahead bounds the line length so legitimate body sentences ending with a "Place, Region" phrase (typically much longer) aren't absorbed. Position-gated to front-matter zone via the outer strip's 8000-char cutoff.
+
+**File:** `docpluck/normalize.py::_ORPHAN_AFFIL_WRAP_TAIL`. Regression tests in `tests/test_normalize_metadata_leak_real_pdf.py` covering positive variants ("Berkeley, CA.", "Cambridge, MA 02138.", etc.) and negative shapes ("(Miller & Prentice, 1994).", citations, body sentences containing place names).
+
 ## CHANGELOG-documented public-API names must be in `__all__` (caught 2026-05-07, v2.0.0 release)
 
 **What:** v2.0.0 CHANGELOG line "`Cell, Table, Figure, StructuredResult` TypedDicts and `TABLE_EXTRACTION_VERSION` re-exported from top-level `docpluck`" was inaccurate — `Cell` was importable via `docpluck.tables.Cell` but not from top-level `docpluck`. Caught by /ship Phase 3 cleanup against `docpluck.__all__`.
@@ -502,3 +532,14 @@ Rotation picks `pool[(N mod L) : (N mod L) + rotation_size]` wrapping. Over `cei
 
 **Bundling N distinct mechanisms in one cycle is OK when each is independently revertible** — the discipline rule "one class of defect per cycle" exists to prevent un-revertible co-fixes. When each fix is a contiguous block touching a different code path with its own test, bundling halves the release/deploy cost vs. N separate cycles. Cycle 14 packaged 3 fixes; cycle 15 packaged 5. Both shipped clean.
 
+
+## 2026-06-07 — finding-key identity must be line-number-invariant; deferred half-fixes resurface
+- A canary-audit `finding_key` that embeds a line number is brittle: any normalize/render change that adds or removes a line shifts every downstream finding's line number, so `--gate-new-only` re-opens KNOWN deferred findings as NEW and falsely blocks the commit. Identity must be severity + verbatim excerpt (the document text is render-stable). Fixed in `_shared/iterate-loop/canary_findings.py::_norm_location`.
+- The Sonnet canary auditor needs an `allowed_omissions` list (front-matter masthead, publication-history dates, running headers) or it flags docpluck's by-design strips as TEXT-LOSS and forces SKIP_CANARY. Mirror the in-session verifier's allowed-omissions list. Wired via `canary.json::verification_protocol.allowed_omissions` → payload → `audit-subagent-prompt.md`.
+- A docstring that says "a separate pass can rejoin/fix this later" is a latent finding, not a note. `_demote_continuation_promoted_headings` deferred its rejoin in cycle 3; it surfaced as canary finding #2 in cycle 5. Finish the fix when you write it.
+- The per-commit canary hook on a paper with a LARGE open deferred surface + single-audit non-determinism will keep requiring SKIP_CANARY until that surface is fully baselined or fixed. Don't read SKIP_CANARY as "gate is broken" — read it as "this canary has known multi-session debt." Surface it; use `--full` (double-audit union) to baseline more completely.
+
+## 2026-06-07 — O5 reference reading-order inversion + a broken WIP that silently disabled column-correction
+- **A handoff item lumped into a "won't-fix" class can be a real, different-class bug.** scimeto's O5 ("chen refs stranded before the References header") was tagged "docpluck won't-fix" by association with the A–D pdftotext-glyph classes. It was actually a *reading-order* defect docpluck owns (pdftotext inverts a banded page's two reference columns). Always re-derive layer-of-origin per item; don't inherit a sibling's disposition.
+- **`except Exception: pass` around an optional pipeline stage hides a hard syntax error.** A prior session left `extract_columns.py` with an IndentationError + a call to an undefined `_detect_2col_midline_gutter` in the working tree. Because `extract.py` wraps the column-correction call in a bare `except Exception: pass` ("signal-only, never block extraction"), the module simply failed to import on every flagged page → column-correction was silently OFF corpus-wide, with no error surfaced. When you see a broad `except: pass` around a feature, test that the feature actually still runs — don't assume "no error" means "working." (Companion to `feedback_no_silent_optional_deps`.)
+- **The safe way to ship a reading-order change (which char-ratio/Jaccard gates are BLIND to): a word-preservation guard.** Accept a geometric re-extraction only if it preserves the page's substantial-word multiset (alphabetic tokens len≥2) — a pure reorder can't drop or fabricate text (rules 0a/0b), and the corpus char-ratio gate (blind to order) still confirms no content change. Confine the new path behind an explicit flag so the legacy path stays byte-identical, and verify the *trigger* fires on a tiny, hand-checked subset (2 of 101 papers here) before trusting it.

{docpluck-2.4.76 → docpluck-2.4.80}/.claude/skills/docpluck-iterate/LEARNINGS.md

@@ -1013,3 +1013,40 @@ Mid-run (between cycle 6 and cycle 7) the user re-stated the LEAVE NOTHING BEHIN
 - Cycle 11 — diagnose remaining individual non-idempotent cases (verify_out scan post-cycle-10 expected ~10-15 from initial 40). Many are likely whitespace-only or single-paper quirks.
 - Group B / B1 — plos-med-1 TABLE-builder cluster. ARCHITECTURAL — needs user direction before coding.
 - Phase 8 Tier-3 verify — confirm v2.4.62 on Railway after this cycle's auto-bump deploys.
+
+---
+
+## Cycle 5 (2026-06-07) — canary findings #1 + #2 cleared (v2.4.79) + canary-gate substrate hardening
+
+**Target (from 2026-06-07 handoff):** clear ip_feldman canary finding #1 (METADATA-LEAK, residual `Received …; revision accepted …` line) and #2 (HALLUCINATION, `Supplemental Materials` fragment). Both DONE + verified.
+
+**Finding #1** — US-format publication-history line. Root cause: every existing `Received…` strip pattern is European-order (`Received DD Month YYYY`); ip_feldman is US `Received Month DD, YYYY; revision accepted …`. Fix: one `_PAGE_FOOTER_LINE_PATTERNS` entry whose date sub-pattern accepts BOTH orders, anchored on the distinctive `revision accepted` + trailing year. The handoff suggested `^Received .*; revision accepted .*$` — I tightened it (verified it rejects body prose like "We received … revision accepted by reviewers in 2025").
+
+**Finding #2** — confirmed **audit FALSE-POSITIVE** (the sentence is real, gold line 86). The real defect was a demote-miss: `_demote_continuation_promoted_headings` stripped a `## Supplemental Materials` false-heading but left it as an orphan bare line (period stripped by the promotion, blank-line padding retained) → read as a hallucinated fragment. Fix: rejoin the demoted continuation to the prior line it grammatically continues + restore the terminal period. The function's own docstring had *deferred* this rejoin ("for now we just stop the false-heading") — a deferred half-fix that surfaced as a finding two sessions later. **Lesson: a docstring that says "a separate pass can do X later" is a latent finding; finish it.**
+
+**Substrate defects found + fixed while gating (LEAVE NOTHING BEHIND):**
+1. **`finding_key` was line-number-sensitive.** `_norm_location` folded a location to `"line N"` keeping the number. My fix shifted ip_feldman's line numbers (removed 1 line, rejoined 1) → deferred Table-10 finding moved `line 1351`→`line 1348` → the `--gate-new-only` ledger saw it as NEW and BLOCKED the commit. This defeats the entire regression-only gate: ANY render change trips it. Fixed `_norm_location` to drop the line number (identity = severity + verbatim excerpt). + invariance self-test. **This is the #1 reason the per-commit canary hook was unusable.**
+2. **Audit had no `allowed_omissions` concept.** The Sonnet canary auditor flagged docpluck's by-design front-matter masthead strip (authors/affiliations/journal/DOI) as TEXT-LOSS — a recurring false-positive that forces SKIP_CANARY. Added an optional `allowed_omissions` payload field (wired through canary-audit.sh + audit-subagent-prompt.md + canary.json) so by-design omissions aren't findings. Mirrors the in-session verifier's allowed-omissions list.
+3. **Single-audit non-determinism vs a large deferred surface.** ip_feldman has a big KNOWN-deferred defect set (Camelot tables + R4 reading-order). The `--quick` single audit samples a DIFFERENT ~7 of them each run, so `--gate-new-only` keeps finding "new" ones until the whole surface is baselined. The `--full` double-audit+union mitigates but can't eliminate. **Practical consequence: per-commit canary on a paper with a large open deferred surface will keep requiring SKIP_CANARY until the deferred work lands — surface this, don't pretend the gate is clean.**
+
+**Gate honesty:** iterate-gate --cycle 1 = FAIL (I2: only audited the litmus; I3: ip_feldman still FAIL on deferred findings). Correct per rule 0e-bis — the cycle made progress but the canary isn't clean. NOT marked PASS, NOT tagged. The 10 deferred findings (Camelot B4 + R4 reading-order) are each their own multi-session architecture effort — surfaced to the user, not bundled.
+
+**Verification:** full pytest 1870 passed; 26/26 corpus baseline; substrate self-tests 17/17; canary audit confirmed zero METADATA-LEAK/HALLUCINATION remain on ip_feldman.
+
+---
+
+## Run: 2026-06-07 (resume) · O5 reference reading-order inversion · v2.4.79 → v2.4.80
+
+### Outcome
+- SHIPPED (uncommitted, user to review) — O5 from the citationguard-iterate handoff: chen + jamison reference lists were reading-order-inverted by pdftotext (a banded page stacks a CRediT table above a 2-column reference list; pdftotext emits the right reference column before the left column carrying the `References` heading → a block of entries stranded above the heading). Fixed via region-aware column re-extraction. chen: 0 stranded / 101 ordered (was 36+ stranded); jamison: 0 / 37; 99 other corpus papers unchanged.
+
+### What the handoff started as vs what it became
+- Began as "address the A–D text-extraction handoff" → user decided **won't-fix, alert scimeto** (A/B superscript flatten+glue, C semantic hyphen, D ref mangling are all pdftotext text-channel / source-PDF — verified at the raw level; Strömwall was already resolved, a pymupdf-fixture artifact). Then the user surfaced **O5** (in `TRIAGE_iterate_2026-06-07.md`, not the A–D handoff) and reversed to "review/verify/fix it all + build O5."
+- **Lesson: don't inherit a sibling item's disposition.** scimeto had tagged O5 "docpluck won't-fix" by association with A–D. Re-deriving layer-of-origin showed O5 is a *reading-order* bug docpluck owns — a different class entirely.
+
+### Blind spots / process
+- **A broad `except Exception: pass` hid a hard import error.** The working tree had a broken in-flight WIP of this exact feature (`extract_columns.py` IndentationError + undefined `_detect_2col_midline_gutter`); the `except: pass` in `extract.py`'s column-correction call meant the module silently failed to import on every flagged page → column-correction was OFF corpus-wide with no surfaced error. Restored to clean HEAD before building. When a feature is wrapped in `except: pass`, test that it actually runs.
+- **The corpus char-ratio/Jaccard gate is blind to reordering** (known: `feedback_ai_verification_mandatory`). The safe shipping pattern for a reading-order change: a **word-preservation guard** (accept the re-extraction only if it preserves the substantial-word multiset — a pure reorder, rules 0a/0b) + confine the new path behind an explicit flag (legacy byte-identical) + verify the trigger fires on a tiny hand-checked subset (2 of 101) before trusting it.
+
+### Tests
+- `tests/test_o5_reference_inversion_real_pdf.py` — real-PDF, parametrized over chen + jamison: detection fires, refs corrected (0 before / ≥20 after header), text preserved (anchor surnames present), detector selective (does NOT fire on a normally-ordered paper).

{docpluck-2.4.76 → docpluck-2.4.80}/.claude/skills/docpluck-iterate/references/ai-full-doc-verify.md

@@ -144,8 +144,11 @@ CHECKS (in order; each must pass):
    the library may apply: U+2212→hyphen, soft-hyphen rejoin, line-wrap reflow,
    NFKC composition, smart-quote→straight-quote. Allowed omissions ONLY:
    running headers, page numbers, copyright lines, Crossref boilerplate,
-   watermark strips, ORCID lines, DOI banner lines. Anything else missing is a
-   TEXT-LOSS finding.
+   watermark strips, ORCID lines, DOI banner lines, publication-history date
+   lines (e.g. "Received <date>; revision accepted <date>", "Received/Revised/
+   Accepted" mastheads — docpluck strips these as journal furniture by long-
+   established design; not sentence-shaped body prose). Anything else missing is
+   a TEXT-LOSS finding.
 
 2. HALLUCINATION check.
    For every substantive paragraph in the rendered .md (≥60 chars,

{docpluck-2.4.76 → docpluck-2.4.80}/.gitignore

@@ -47,3 +47,9 @@ tmp_*.py
 
 # Verification harness outputs (scripts/harness) — extraction artifacts, not committed.
 verify_out/
+
+# Canary-audit persistent open-finding ledger (per-machine cross-run audit
+# STATE written by ~/.claude/skills/_shared/iterate-loop/canary-audit.sh). It is
+# git-SHA-keyed but reflects local audit history, so it's machine-local state —
+# not committed (canary.json, the sibling CONFIG, IS tracked).
+.claude/skills/_project/canary-findings-ledger.json

{docpluck-2.4.76 → docpluck-2.4.80}/CHANGELOG.md

@@ -1,5 +1,57 @@
 # Changelog
 
+## [2.4.80] — 2026-06-07
+
+**O5 — reference reading-order inversion fixed (citationguard-iterate handoff).** Region-aware column re-extraction (`extract.py` + `extract_columns.py`); no `NORMALIZATION_VERSION` change (text channel only). Keyed on a STRUCTURAL signature — reference-list entries serialized ABOVE their own `References` heading on a page — never paper identity. Gated against the 26-paper corpus baseline + the column-test suite (32 existing pass) + 5 new real-PDF tests.
+
+- **Root cause:** on two-column papers whose final page stacks a full-width contributor (CRediT) table ABOVE a two-column reference list, pdftotext serializes the page's columns out of order — emitting the RIGHT reference column before the LEFT column that carries the `References` heading. A block of reference entries is thereby stranded ABOVE the heading, invisible to any consumer scanning for references after it (citationguard-iterate O5: "36 chen refs stranded before the References header"). This is the R4/reading-order half of the same column-interleave root cause diagnosed for `ip_feldman` (finding #5 in 2.4.79).
+- **`_detect_reference_inversion_pages`** (`extract_columns.py`): a cheap, text-only detector flagging a page when ≥3 reference-list-entry lines (`Surname, F. M.` shape, anchored at line start so in-text citations don't match) appear before that page's `References`/`Bibliography` heading. Reference entries cannot precede their own section heading in correct reading order, so this is an unambiguous inversion signature. Fires on exactly 2 of the 101-paper corpus (`chen_2021_jesp` p19, `jamison_2020_jesp` p9) — both genuine inversions.
+- **`_detect_2col_midline_gutter`** (`extract_columns.py`): a full-height empty central GUTTER-STRIP midline detector. Stronger than the word-center histogram for narrow (~4pt) reference-column gutters the histogram can't resolve, and — because a clean full-height strip cannot coexist with a full-width table row crossing the center — it lets the prose columns be corrected while BYPASSING the y-row bilateral gate that (correctly) protects genuine table pages. Confined to the inversion path (`allow_gutter_fallback`), so the legacy column-interleave path is byte-identical.
+- **Word-preservation guard** (`splice_column_corrected_pages` `word_preserve_pages`): a flagged page's geometric re-extraction is accepted ONLY if it preserves the page's substantial-word multiset (every alphabetic token of length ≥2) — a pure reorder that can never drop or fabricate reference text (rules 0a / 0b). Trivial digit/single-char churn at crop boundaries is tolerated.
+- **Result:** `chen_2021_jesp` 0 reference entries stranded before the heading (was 36+); 101 entries now follow it in alphabetical order. `jamison_2020_jesp` likewise (0 stranded; 37 entries). All other 99 corpus papers unchanged (detector does not fire; legacy path byte-identical). Classes A–D from the same handoff are pdftotext text-channel / source-PDF artifacts and remain consumer-side (citelink) — see `docs/superpowers/handoffs/2026-06-07-text-extraction-defects-from-citationguard-iterate.md`.
+
+New regression tests: `tests/test_o5_reference_inversion_real_pdf.py` (detection + correction + text-preservation + detector-selectivity; real-PDF, parametrized over chen + jamison).
+
+## [2.4.79] — 2026-06-07
+
+**Cycle 5 — canary findings #1 (METADATA-LEAK) + #2 (HALLUCINATION false-positive) cleared on `ip_feldman_2025_pspb`.** `NORMALIZATION_VERSION` 1.9.27 → 1.9.28. Both fixes keyed on a STRUCTURAL signature, never paper identity; gated against the 26-paper corpus baseline + full pytest.
+
+- **US-format publication-history line strip** (`normalize.py` `_PAGE_FOOTER_LINE_PATTERNS`): a new pattern strips the `Received Month DD, YYYY; revision accepted Month DD, YYYY` line (Sage / APA journals — PSPB) that leaked between `## Keywords` and `## Introduction`. The existing publication-history patterns are all European-order (`Received DD Month YYYY; Accepted …`); this adds the US `Month DD, YYYY` order with the distinctive `revision accepted` phrase. The date sub-pattern accepts either order; the `revision accepted` token plus a trailing year makes the line unmistakably journal boilerplate, never body prose. Clears canary finding #1. (The verifier's allowed-omissions list — `references/ai-full-doc-verify.md` — now records publication-history dates as journal furniture docpluck strips by long-established design, so the finding can't churn METADATA-LEAK→TEXT-LOSS.)
+- **Continuation-demoter rejoin** (`render.py` `_demote_continuation_promoted_headings`): when a `## <Title>` is demoted because the prior line ends in a continuation word (soft-wrap split false-promotion), the demoted phrase is now **rejoined to the prior line it grammatically continues**, with the trailing sentence terminator restored when the phrase completes the sentence — instead of being left as an orphan bare line. An orphaned `Supplemental Materials` fragment (period stripped by the promotion, blank-line padding retained) read as a hallucination to the AI verifier; `The calculated effect sizes are summarized in the` + `Supplemental Materials` now renders as the single continuous sentence `The calculated effect sizes are summarized in the Supplemental Materials.` matching the gold. Clears canary finding #2 (audit false-positive — the text was always real; the defect was the spurious mid-sentence split). This completes the deferred rejoin the function's own docstring had flagged ("a separate normalize pass can rejoin it … for now we just stop the false-heading").
+
+New regression tests: `tests/test_normalize_metadata_leak_real_pdf.py` (US-format Received-line contract + variant + real-PDF cases), `tests/test_hallucinated_heading_continuation_guard.py` (`TestContinuationRejoin` — rejoin/no-orphan, period restoration, no-double-period, lowercase-continuation guard + real-PDF rejoined-sentence assertion).
+
+**Still NOT tagged** — canary retains the deferred multi-session findings: #3 Table 2 mid-Introduction (Camelot/B4), #4 Table 10 mid-External-Analysis (Camelot/B4), #5 `## Discussion` displaced Table-9 footnote + External-Analysis fragment (R4 column-aware reading-order). See the 2026-06-07 handoff.
+
+## [2.4.78] — 2026-06-06
+
+**Cycle 4 redux — canary findings #1/#3/#4 cleared on `ip_feldman_2025_pspb` + citationguard text-extraction handoff Defect 1.** `NORMALIZATION_VERSION` 1.9.26 → 1.9.27. Driven by the Sonnet-via-Claude-Max canary audit (headless gate now operational after `claude setup-token`). Five render/normalize fixes, each keyed on a STRUCTURAL signature (never paper identity), each gated against the 26-paper corpus baseline + full pytest:
+
+- **Cluster E re-land** (`normalize.py` `_FRONTMATTER_LEAK_LINE_PATTERNS`): the `_ARTICLE_TYPE_CODE` (`research-article2025` etc.) + `_BARE_ARTICLE_ID` (6–8 digit standalone) patterns drafted in v2.4.77 were **reverted there** (they exposed a wrapped-title duplicate) and shipped as dead code. Now genuinely wired — the revert's blocker is resolved by the new wrapped-title demoter below. Clears the bare article-ID + article-type-code half of canary finding #1.
+- **Wrapped-title-duplicate demoter** (`render.py` `_demote_wrapped_title_duplicate`): strips a `### {prefix-of-H1}` + continuation block immediately under the H1 (pdftotext emits the title twice on PSPB/Sage column layouts). Token-prefix match vs the H1 with a ≥75%-coverage gate; runs after title rescue. Resolves the side-effect that forced the v2.4.77 Cluster E revert.
+- **Cohesive masthead-block strip** (`render.py` `_strip_frontmatter_masthead_block`): strips the residual publisher masthead (author+superscript lines, journal-name wraps, page range, copyright tail, `DOI:` label, bare DOI) between the H1 and the first `## ` heading. Self-limiting ≥2-hard-marker gate; a prose-break guard preserves an undetected abstract. Clears the remainder of canary finding #1 — the doc now flows `# Title` → `## Abstract` cleanly.
+- **Column-wrapped heading repair** (`render.py` `_repair_column_wrapped_headings`): Rule A promotes a body `{Title} et al.` + bare `(YYYY)` citation-wrap to `### {Title} et al. (YYYY)` (finding #3 — "Choice of Study for Replication"); Rule B reattaches a short colon-/paren-led orphan tail onto a heading split mid-title (finding #4 — "Original Hypotheses…Target Article: Jordan et al. (2011)"). Both gated on followed-by-prose so two real sibling headings are never merged.
+- **Soft-hyphen line-break dehyphenation** (`normalize.py` S6): a U+00AD immediately before a line break is always a discretionary extraction hyphen splitting one word across the wrap; the join now drops the U+00AD *and* the newline (gated on a following letter) BEFORE the existing bare strip, so `relation­\nship` → `relationship` instead of the previously-surviving space-broken `relation ship`. Closes citationguard text-extraction handoff Defect 1 (recovered com/mitment, pro/motion, altru/ism, relation/ship on chan_feldman_2025_cogemo; U+00AD count 0). Defect 3 (dropped reference line) verified CLEAN in docpluck (pymupdf-only); Defect 2 (position-dependent glyph mis-map `Västfjäll`→`Vastfall`) reproduced but deferred — needs a same-document surname-consensus normalizer (architecture decision).
+
+**Run-11 hallucinated-heading guards** (`render.py` `_promote_isolated_titlecase_subsection_headings` + new `_strip_pre_title_heading_noise` / `_nearest_h2_parent_label`): three general structural guards that clear four mis-promoted-heading findings across the canary set:
+
+- **Pre-H1 heading strip** — a heading sitting ABOVE the document title is a journal-section label promoted in the masthead zone; stripped after title rescue. Clears `### FlashReport` (ar_apa); generalizes to `### RESEARCH ARTICLE`-shape front matter.
+- **Regular-path blacklisted-parent reject** — the regular promotion path now consults `_CHAIN_REJECT_PARENTS` (walking back to the nearest `## ` parent, tolerating interleaved running-header lines), matching the chain path. Clears `### Methodology` (a CRediT role label under `## Author Contributions` on plos_med, promoted when a running-header broke chain adjacency).
+- **Lowercase-body reject** — a candidate whose following body's first alphabetic char is lowercase is a fragment torn from a running sentence, not a title. Clears `### Close replication` (chan_feldman) and `### Proced` (plos_med truncation fragment).
+
+Deferred (surfaced, not hacked): `### Reasons for change` (ip_feldman — a Table 5 column header; needs table-region awareness), the `## Data Availability` end-matter absence (RCA correction: it never enters the text channel — pdftotext drops the box; cross-channel/layout recovery, NOT a demoter over-strip as the run-11 handoff assumed), and the citationguard `Open Science Collaboration`→`Open, S. C.` org-author collapse (RCA: baked into the PDF's embedded text by the publisher; identical in pdftotext AND pdfplumber; no safe general docpluck fix — routed to CitationGuard for DOI/CrossRef author reconciliation).
+
+New regression tests: `tests/test_render_frontmatter_masthead.py` (27 cases — wrapped-title demoter, masthead-block gate, column-wrapped heading repair, pre-H1/blacklist-parent/lowercase-body guards + real-PDF), `tests/test_normalize_soft_hyphen_dehyphenation.py` (5 cases incl. real-PDF), plus front-matter cases added to `tests/test_normalize_metadata_leak_real_pdf.py`. **NOT yet tagged** — canary still has open table findings (#2/#6/#7 Camelot, multi-session) + reading-order findings (#5/#8, R4 column-aware) + the deferred items above. See the 2026-06-06 handoff.
+
+## [2.4.77] — 2026-05-26
+
+**Cluster E front-matter cleanup follow-up to v2.4.76.** `NORMALIZATION_VERSION` 1.9.25 → 1.9.26. Three additional publisher-metadata strip patterns observed after v2.4.76 shipped (Stream A continuation work for ip_feldman_2025_pspb + ar_apa front-matter):
+
+- **`_PAGE_FOOTER_LINE_PATTERNS`** (`normalize.py`): new `^Article reuse guidelines:?$` pattern. Sage / PSPB publisher boilerplate that pdftotext emits as a standalone front-matter line. Tight-anchored so it can't match body prose.
+- **`_FRONTMATTER_LEAK_LINE_PATTERNS`** (`normalize.py`): new `_ARTICLE_TYPE_CODE` and `_BARE_ARTICLE_ID` patterns. The article-type code pattern matches `research-article2025`, `meta-analysis2024`, etc. (publisher-internal article-type slug + year). The bare-article-ID pattern matches a standalone 6–8 digit line (the DOI's last segment repeated alone at top-of-doc). Both are position-gated to the front-matter zone (first 8000 chars or 1/6 of doc) via the existing `_strip_frontmatter_metadata_leaks` infrastructure — body false positives impossible.
+
+Verification: `test_ip_feldman_top_of_doc_cleaned_real_pdf` PASS in isolation and in 64-test batch. No regression on the v2.4.76 corpus.
+
 ## [2.4.76] — 2026-05-25
 
 **§A R4 column-aware re-extraction LANDED — closes jama-open-1 D4 (Key Points sidebar missing).** `NORMALIZATION_VERSION` 1.9.24 → 1.9.25 (concurrent with EC-T1's bump). Closes the final defect of the 2026-05-25 Haiku-orchestration pretest jama-open-1 cluster (HANDOFF_2026-05-25_pretest-followups.md Issue 1 — 5 of 5 defects now closed).

{docpluck-2.4.76 → docpluck-2.4.80}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.76
+Version: 2.4.80
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://docpluck.app
 Project-URL: Documentation, https://docpluck.app/api-docs
@@ -35,6 +35,7 @@ Requires-Dist: httpx>=0.27; extra == 'dev'
 Requires-Dist: lxml>=5.0.0; extra == 'dev'
 Requires-Dist: mammoth>=1.8.0; extra == 'dev'
 Requires-Dist: pytest-anyio>=0.0.0; extra == 'dev'
+Requires-Dist: pytest-xdist>=3; extra == 'dev'
 Requires-Dist: pytest>=8; extra == 'dev'
 Requires-Dist: python-docx>=1.0.0; extra == 'dev'
 Requires-Dist: rapidfuzz>=3; extra == 'dev'

{docpluck-2.4.76 → docpluck-2.4.80}/TODO.md

@@ -131,6 +131,29 @@ Add only when a real downstream consumer asks for one. YAGNI until then.
 - [ ] **Task #8**: `ai-gold.py resolve` should accept stem names + source-PDF paths (or docs redirect to `check`).
 - [ ] **Task #9**: `ai-gold.py onboard` needs `--skip-legacy` / `--ignore-unresolvable` flag (citationguard onboard halted on 3,018 legacy bare-stem keys).
 
+## 2026-06-07 — v2.4.78 landed (committed 73462e3, unpushed→pushed, NOT tagged)
+
+> **Source:** Cycle-4-redux session. See `docs/superpowers/handoffs/2026-06-07-v2.4.78-landed-canary-iterate.md` for the full state + next-session opener. v2.4.78 cleared canary findings #1/#3/#4 + 4 run-11 hallucinated-heading findings + citationguard soft-hyphen Defect 1. Corpus 26/26, full pytest 1861 passed, canary re-audit 8→5.
+
+### Remaining canary findings on ip_feldman_2025_pspb (from 2026-06-07 re-audit @ 73462e3)
+
+- [x] **METADATA-LEAK** — ✅ CLEARED in **v2.4.79** (cycle 5). New `_PAGE_FOOTER_LINE_PATTERNS` entry strips US-format `Received Month DD, YYYY; revision accepted Month DD, YYYY` (date sub-pattern accepts either order; gated on `revision accepted` + trailing year). The handoff-suggested `^Received .*; revision accepted .*$` was tightened to avoid matching body prose. Real-PDF + contract tests added.
+- [x] **HALLUCINATION** — ✅ CLEARED in **v2.4.79** (cycle 5). Confirmed **audit false-positive** (the sentence is real — gold line 86); the actual defect was a spurious mid-sentence split + dropped period left by `_demote_continuation_promoted_headings`. The demoter now rejoins the demoted continuation to the prior line it continues + restores the terminal period. Verifier allowed-omissions doc updated so publication-history dates can't churn METADATA-LEAK→TEXT-LOSS.
+- [ ] **TABLE #3 + #4 (B4) AND SECTION-BOUNDARY #5 (R4) share ONE root cause** — DIAGNOSED 2026-06-07 (cycle 5), see [`docs/superpowers/specs/2026-06-07-ip_feldman-B4-R4-column-interleave-diagnosis.md`](docs/superpowers/specs/2026-06-07-ip_feldman-B4-R4-column-interleave-diagnosis.md). pdftotext column-interleaves table-bearing two-column pages, interleaving table caption/headers/cells AND real body prose into one stream. **Empirically NOT single-cycle** (corrects optimistic subagent estimates): the existing R4 infra's bilateral gate (protects table pages corpus-wide) rejects the target pages (measured bilateral 0.37–0.44 > 0.30), and the B4 leak can't be stripped render-side because real prose is interwoven with table content (would risk TEXT-LOSS, rule 0a). **Real fix = region-aware column detection** (segregate full-width table band from 2-col prose band before column-correcting) — multi-session architecture, **needs user go-ahead**.
+
+### Deferred from this session (surfaced, not hacked)
+
+- [ ] **`### Reasons for change`** (ip_feldman) — Table 5 column header promoted to heading; needs table-region awareness (the body-coherence guard doesn't catch it because its body starts capitalized). RCA: rank-3 in the 2026-06-06 run-11 RCA.
+- [ ] **`## Data Availability` end-matter absent** — RCA CORRECTED the run-11 "demoter over-strip" premise: the section never enters the text channel (pdftotext drops the title-page box). Needs cross-channel (pdfplumber) recovery, same architecture class as B7. NOT a demoter exception.
+- [ ] **Glyph `Västfjäll`→`Vastfall`** (ar_apa/collabra, citationguard Defect 2) — baked pdftotext CID-font mis-map; needs a same-document surname-consensus normalizer (new subsystem). **Product/architecture decision on scope.**
+- [ ] **org-author `Open Science Collaboration`→`Open, S. C.`** — baked into the PDF's embedded text by the publisher (identical in pdftotext AND pdfplumber); no safe general docpluck fix. **Routed to CitationGuard** (DOI/CrossRef author reconciliation).
+- [ ] **Tag v2.4.78** once the full canary set clears (currently 5 open). Then bump `PDFextractor/service/requirements.txt` pin + run `/docpluck-deploy`.
+
+### Substrate / infra follow-ups (2026-06-07)
+
+- [x] **Tune the canary git-hooks to be ledger-aware (regression-only gate).** ✅ DONE (2026-06-07 cycle 5). `canary-audit.sh --gate-new-only` mode had landed but the hooks weren't passing it. Now wired: `.git/hooks/pre-commit` runs `--quick … --gate-new-only` (routine commits block only on NEW/regressed findings, not the deferred baseline); `.git/hooks/pre-push` uses `--gate-new-only` for **main** pushes but keeps the **strict** no-exceptions gate for **tag** pushes (a release tag must ship a fully-clean canary). Substrate self-tests 16/16 pass.
+- [ ] **Modern Standby permanent disable** (optional, user/admin): `reg add HKLM\SYSTEM\CurrentControlSet\Control\Power /v PlatformAoAcOverride /t REG_DWORD /d 0 /f` + reboot. Current `powercfg /change standby-timeout-ac 0` works but can revert on power-plan change. See memory `feedback_long_runs_die_on_this_machine`.
+
 ### Replicate canary-audit pattern to other iterate skills (after docpluck proven)
 
 - [ ] **escicheck-iterate** — easiest pilot (46 successful phase_5d_runs already, well-defined stats-family defect taxonomy). Update `verification_protocol` in its `canary.json`.

{docpluck-2.4.76 → docpluck-2.4.80}/docpluck/__init__.py

@@ -77,7 +77,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.76"
+__version__ = "2.4.80"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.76 → docpluck-2.4.80}/docpluck/extract.py

@@ -134,22 +134,37 @@ def extract_pdf(pdf_bytes: bytes, *, sections: list[str] | None = None) -> tuple
         # conditional fallback, NOT a default tool swap.
         try:
             from .normalize import _detect_column_interleave_pages
-            from .extract_columns import splice_column_corrected_pages
+            from .extract_columns import (
+                splice_column_corrected_pages,
+                _detect_reference_inversion_pages,
+            )
             ff_offsets: list[int] = [0]
             for idx, ch in enumerate(text):
                 if ch == "\f":
                     ff_offsets.append(idx + 1)
+            # Two cheap text-only detectors decide whether the (more expensive)
+            # layout extraction + geometric re-order is worth running:
+            #  - legacy column-INTERLEAVE (sentences woven between columns), and
+            #  - O5 reading-order INVERSION (a page's reference entries serialized
+            #    ABOVE their own References heading — chen page 19). The inversion
+            #    pages are corrected under the word-preservation guard so the
+            #    reorder can never lose or fabricate text (rule 0a / 0b).
             flagged_pages = _detect_column_interleave_pages(text, tuple(ff_offsets))
-            if flagged_pages:
+            inversion_pages = _detect_reference_inversion_pages(text, tuple(ff_offsets))
+            all_pages = sorted(set(flagged_pages) | set(inversion_pages))
+            if all_pages:
                 from .extract_layout import extract_pdf_layout
                 layout_doc = extract_pdf_layout(pdf_bytes)
+                changed: list[int] = []
                 corrected = splice_column_corrected_pages(
-                    text, layout_doc, ff_offsets, flagged_pages,
+                    text, layout_doc, ff_offsets, all_pages,
                     pdf_bytes=pdf_bytes,
+                    word_preserve_pages=inversion_pages,
+                    changed_out=changed,
                 )
-                if corrected and corrected != text:
+                if corrected and corrected != text and changed:
                     text = corrected
-                    method = f"{method}+column_corrected:{','.join(map(str, flagged_pages))}"
+                    method = f"{method}+column_corrected:{','.join(map(str, changed))}"
         except Exception:
             pass