docpluck 2.4.83__tar.gz → 2.4.85__tar.gz

{docpluck-2.4.83 → docpluck-2.4.85}/.claude/skills/docpluck-deploy/SKILL.md

@@ -267,26 +267,43 @@ curl -s -o /dev/null -w "%{http_code}" https://docpluck.app/login
 ```
 Must return 200.
 
-### 3. Railway Service Health
+### 3. Railway Service Health + deployed-version gate (AUTHORITATIVE)
+
+`/health` is public and reports the docpluck version the live service actually imported — this is the authoritative "the new library is deployed" gate (the auth-gated `/extract` smoke in check 4 cannot run without a `dp_` API key). A clean boot here also proves docpluck imports without error at the new version.
+
 ```bash
-curl -s https://extraction-service-production-d0e5.up.railway.app/health
+LIB_VERSION=$(grep '^__version__' C:/Users/filin/Dropbox/Vibe/MetaScienceTools/docpluck/docpluck/__init__.py | grep -oE '[0-9]+\.[0-9]+\.[0-9]+')
+curl -s --max-time 25 https://extraction-service-production-d0e5.up.railway.app/health | python -c "
+import sys, json
+d = json.load(sys.stdin)
+print('health:', d.get('status'), '| docpluck:', d.get('docpluck_version'), '| db:', d.get('database'))
+assert d.get('status') == 'ok', 'service not ok'
+assert d.get('docpluck_version') == '$LIB_VERSION', f\"deployed docpluck {d.get('docpluck_version')} != released $LIB_VERSION\"
+print('Health + version gate: PASS')
+"
 ```
-Must return `{"status":"ok",...}`.
+**Gate:** `status == ok` AND `docpluck_version == <released version>` AND `database == connected`. A version mismatch means Railway has not finished redeploying on the new pin — wait and re-check; do not declare the deploy done.
+
+### 4. Authenticated extraction smoke (OPTIONAL — requires a `dp_` API key)
+
+`/extract` is auth-gated: an unauthenticated POST returns `401 {"detail":"Missing or invalid API key. Use: Authorization: Bearer dp_..."}`. So this is a real end-to-end extraction check ONLY when a key is available (e.g. `DP_SMOKE_API_KEY` in the env). The check-3 health/version gate is the authoritative deployed-version proof; this is extra confidence on the extraction path. Skip (mark INCONCLUSIVE, not FAIL) when no key is present.
 
-### 4. Smoke Test (if service is live)
 ```bash
-# Test extraction endpoint directly
-curl -s -X POST https://extraction-service-production-d0e5.up.railway.app/extract \
-  -F "file=@test-pdfs/apa/chan_feldman_2025_cogemo.pdf" | python -c "
+if [ -n "$DP_SMOKE_API_KEY" ]; then
+  curl -s --max-time 90 -X POST https://extraction-service-production-d0e5.up.railway.app/extract \
+    -H "Authorization: Bearer $DP_SMOKE_API_KEY" \
+    -F "file=@test-pdfs/apa/chan_feldman_2025_cogemo.pdf" | python -c "
 import sys, json
 data = json.load(sys.stdin)
-print(f'Engine: {data[\"metadata\"][\"engine\"]}')
-print(f'Chars: {data[\"metadata\"][\"chars\"]}')
-print(f'Quality: {data[\"quality\"][\"score\"]}')
-assert data['metadata']['chars'] > 10000, 'Too few chars'
-assert data['quality']['score'] >= 80, 'Quality too low'
-print('Smoke test: PASS')
+m, q = data.get('metadata', {}), data.get('quality', {})
+print('engine:', m.get('engine'), '| chars:', m.get('chars'), '| quality:', q.get('score'))
+assert (m.get('chars') or 0) > 10000, 'Too few chars'
+assert (q.get('score') or 0) >= 80, 'Quality too low'
+print('Authenticated smoke test: PASS')
 "
+else
+  echo 'DP_SMOKE_API_KEY not set — authenticated /extract smoke SKIPPED (INCONCLUSIVE). Health+version gate (check 3) is authoritative.'
+fi
 ```
 
 ### 5. Daily-Digest Dry-Run Smoke (post-deploy, CRITICAL)

{docpluck-2.4.83 → docpluck-2.4.85}/.claude/skills/docpluck-iterate/LEARNINGS.md

@@ -1123,3 +1123,54 @@ Add a permanent corpus test `tests/test_column_splice_preserves_raw_multiset.py`
 
 ### Edge cases
 - Hyphenated initials (`M.-J. O’Brien et al.`) need `[A-Z]\.[-\s]*` (not `[A-Z]\.\s*`) in the initial group, else they slip the shape. Curly apostrophe `’` must be in the surname char-class.
+
+---
+
+## 2026-06-12 — CitationGuard handoff 2026-06-10 (4 items): 1 fix + 3 won't-fix
+
+**Target:** verify+fix the 4 text-extraction defects filed in
+`CitationGuard/docs/DOCPLUCK_HANDOFF_2026-06-10.md`. Shipped v2.4.84
+(NORMALIZATION_VERSION 1.9.31).
+
+**Methodology lesson (the load-bearing one): VERIFY THE LAYER before
+dispositioning a consumer-filed "extraction defect".** A CitationGuard handoff
+calls everything a "docpluck text-extraction defect", but the layer differs and
+determines whether it's ours:
+- Reproduce each defect at HEAD against the *exact view the consumer reads*
+  (CitationGuard consumes `/extract?normalize=academic` → `normalize_text(raw,
+  academic)`), AND against *raw pdftotext*.
+- If the token is **present in raw pdftotext but absent in academic-normalized →
+  it's a DOCPLUCK normalize bug** (item 4: "of 3 instruments" → R2 deleted "3").
+- If the token is **already corrupted in raw pdftotext → check pdfplumber too**.
+  Corrupted identically in BOTH MIT extractors ⇒ PDF-embedded-text-layer loss,
+  no in-text signal, no general fix, tool-swap forbidden ⇒ won't-fix (items 1–3).
+Do NOT assume "filed by citationguard ⇒ pdftotext-layer won't-fix" — that
+generalization (memory `project_citationguard_extraction_defects_wontfix`) is
+true for SOME items, but item 4 was squarely ours. The 5-min reproduce-at-HEAD
+step (top preflight lesson) is what separated them.
+
+**The fix (general, not whack-a-mole):** R2 (references page-number scrub) was
+deleting any digit whose value appears as a standalone page-number line when it
+sat between two lowercase words — so the quantifier "3 instruments" (page 3 of
+the PDF) was stripped. The pre-existing guard was an *open-ended noun allowlist*
+(`years|participants|…`) that v2.4.17 had been extending one noun at a time.
+Replaced the strategy with a **closed-class signal**: a page leak follows a
+CONTENT word ("psychological 41 science"); a quantifier follows a FUNCTION word
+("of 3 instruments"). Closed classes generalize where open allowlists can't.
+Kept the guard **purely additive** (only preserves, never strips more) so it
+cannot regress the must-strip cases — the safe direction given rule 0a.
+
+**Verification without the harness:** camelot + `claude` CLI were both absent on
+this machine, so the full Phase-5H harness / canary-audit couldn't run. The
+substitute that actually proves a normalize-only, additive change is a
+**deterministic 101-PDF old-vs-new academic-normalized diff**: stash the change,
+regenerate all outputs, `diff` per paper, and READ every changed line. Result: 5
+papers changed, all correct restorations, 0 regressions — and it surfaced 4
+*pre-existing* silent digit-drops the fix also repaired (amp_1, bmc_med_3,
+ieee_access_5 ×2, maier_2023). This diff IS the canary evidence; recorded it as
+the `SKIP_CANARY=1` justification when the `claude`-less pre-commit hook hard-failed.
+
+**Process miss to fix next time:** no working docpluck venv on this machine
+(pdfplumber/camelot/pytest all had to be pip-installed mid-run into C:\Python314).
+Stand up a proper env before an iterate run so the harness + camelot table tests
+aren't skipped.

{docpluck-2.4.83 → docpluck-2.4.85}/.github/workflows/test.yml

@@ -22,7 +22,14 @@ jobs:
           python-version: ${{ matrix.python-version }}
 
       - name: Install poppler-utils (for pdftotext)
-        run: sudo apt-get install -y poppler-utils
+        # `apt-get update` first: the runner image ships a stale package list
+        # that pins a poppler point-release (e.g. 24.02.0-1ubuntu9.8) which the
+        # mirror has since superseded, so a bare `install` 404s on the old .deb.
+        # Refreshing the lists pulls the current version. See the v2.4.84 CI
+        # flake (2026-06-12): "Failed to fetch libpoppler134 … 404 Not Found".
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y poppler-utils
 
       - name: Install dependencies
         run: pip install -e ".[dev]"

{docpluck-2.4.83 → docpluck-2.4.85}/CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## [2.4.85] — 2026-06-12
+
+**Harvard name-year reference splitting (D1) + page-break reference stitch & category-label running-header strip (D2).** `NORMALIZATION_VERSION` 1.9.31 → 1.9.32.
+
+Surfaced by `CitationGuard/docs/DOCPLUCK_HANDOFF_2026-06-12.md`.
+
+**D1 — HIGH IMPACT — Harvard / Cambridge bibliographies collapsed into one paragraph.** R3 (the references-span continuation join) keeps each reference on its own logical line by breaking whenever a line *looks like a reference start*. That detection (`_looks_like_ref_start`) recognised Vancouver (`12. Surname`), IEEE (`[12] Surname`), and APA (`Surname, A.` — note the comma after the surname) — but **not** the Harvard / Cambridge name-year form `Surname A and Surname B (2020) …`, which has *no* comma between surname and initials. So on `bjps_1` (British Journal of Political Science, 109 entries) R3 treated every entry boundary as a mid-entry wrap and joined the **entire** reference section onto a single line. Downstream this made the consumer (citelink) parse 9 of 109 references (refs.f1 0.051).
+
+The fix adds `_REF_START_HARVARD`, keyed on the structural signature *author-block + parenthesised 4-digit year*: a Title-case surname (Latin-Extended, so `Häusermann` / `Tuğal` qualify; hyphenated and compound surnames and `van der` / `Van der` particles handled) + 1–4 initials (spaced `R J`, glued `DH`, hyphenated `H-G`, or period `R.`), optionally chained with ` and `/`&`/comma or `et al.` (incl. `Surname K, et al.`), an optional `(eds)` editor marker, terminated by `(YYYY)`. The parenthesised year is the strong anchor that keeps mid-entry wrap lines (`American Journal of\nPolitical Science 64, 904-20.`) from matching. The same signature is added to `_find_references_spans` so a *pure*-Harvard bibliography (no numbered/IEEE/APA entries at all) is still detected as a references span and R3 runs. Result on `bjps_1`: 109 entries split one-per-line, **zero** entries collapsed; across the 8-paper BJPS Harvard corpus + 9 Nature papers, ≥95% of entries split correctly (was: every Harvard bibliography fully collapsed).
+
+**Same-class fix surfaced by the D1 broad-read — APA ref-start missed accented / particle / compound surnames.** `_REF_START_APA` used an ASCII-only `[A-Z][a-z]+` surname, so `Yücel, M.`, `de Kovel, C.`, and `Karlsson Linnér, R.` were not recognised as entry starts and merged into the preceding reference (nat_comms_5: 10 collapsed pairs; nathumbeh_2: several). It now reuses the shared Latin-Extended surname block (with particles and an optional compound second word), keeping the comma-after-surname APA discriminator. Result: nat_comms_5 collapsed-pairs 10 → 1, nathumbeh_2 6 → 3 (the 3 remaining are supplementary-section prose, not reference merges). The change is a strict widening of a tight `Surname, Initial.` signature — journal-tail continuation lines (`Developmental psychology, 50(12), …`) still do not match.
+
+**D2 — running-header injected mid-reference + page-break-orphaned year.** On `nat_comms_2` (Nature Communications) ref 34 straddles a page break; pdftotext emits `…EAE based on␊␊␌Article␊histology, … (2008).`. The category label `Article` (Nature prints it atop every page) survived — `_looks_like_running_header_or_footer` recognised multi-word banners and author-pair headers but not a bare single-word category label — and welded into the entry, while the page-break blank line left the entry's `(2008)` year in a detached paragraph (citelink parsed ref 34 with an empty year). Two coordinated fixes:
+
+1. `_CATEGORY_LABEL_HEADER` (new shape in `_looks_like_running_header_or_footer`) recognises the publisher article-type labels H0 already curates (`Article`, `Review`, `Letter`, `Matters Arising`, `Original Investigation`, …). Combined with P0r's existing ≥3-standalone-repetition guard this strips the recurring label wherever it appears (including mid-references), so a one-off body occurrence is never touched. Bare common words (`Research`, `Comment`) are excluded to avoid colliding with section headings.
+2. R3 now **bridges a page-break blank line** when the current entry is syntactically *incomplete* (does not end in sentence-terminal punctuation) — a page-break split leaves the head mid-clause (`…EAE based on`), so the tail rejoins; a *completed* entry (`…46, 215-39.`) followed by a blank is the end of the list, so a post-reference trailer (`Cite this article: …`) is **not** absorbed. A form-feed stitch (`\f` → newline inside the span) handles the case where the page header survives upstream.
+
+Result on `nat_comms_2`: ref 34 is a single logical line carrying its `(2008)` year, with no `Article` injection.
+
+New regression tests in `tests/test_harvard_refs_pagebreak_stitch.py`: Harvard ref-start positives (glued/hyphenated/accented/compound/particle surnames) and continuation-line negatives; the bjps_1 one-per-line and nat_comms_2 year-recovery defects at the text level; the page-break stitch + header strip; the trailer-not-absorbed guard; and real-PDF manifest-with-skip assertions on both fixtures.
+
+## [2.4.84] — 2026-06-12
+
+**R2 page-number scrub — general quantifier-head guard (stop deleting digits from reference titles).** `NORMALIZATION_VERSION` 1.9.30 → 1.9.31.
+
+Surfaced by `CitationGuard/docs/DOCPLUCK_HANDOFF_2026-06-10.md` (item 4): the `plos_med_1` reference "Clinimetric properties of **3** instruments measuring postoperative recovery…" rendered as "…properties of instruments measuring…" — the standalone "3" was silently deleted. Same class as the earlier Mayiwar "3"-drop. Root cause: R2 (the references-span page-number scrub) strips any digit whose value also appears as a standalone page-number line, when it sits between two lowercase words. "3" is page 3 of the PDF, so the legitimate quantifier "3 instruments" was stripped. The v2.4.17 body-noun allowlist (`years|participants|…`) is necessarily incomplete and never enumerated "instruments" — adding nouns one at a time is whack-a-mole.
+
+The fix keys on a **closed class** instead: a genuine page-number leak ("psychological **41** science", "recovery **12** in a population") follows a *content* word, whereas a legitimate quantifier ("of **3** instruments", "the **5** factors", "and **3** in Vastfall") follows a *function* word (article / preposition / determiner — a finite closed set). `_r2_is_body_phrase` now also preserves a digit when the immediately-preceding token is in `_R2_QUANTIFIER_HEAD_WORDS`. The guard is purely **additive** — it only ever *preserves* a digit, never strips one — so it cannot make R2 delete anything it didn't already, honoring the correctness asymmetry (silently dropping a digit from a reference title, rule 0a, is far worse than leaving a stray page number). The body-noun list also gained the obvious research-countable nouns (`instruments|measures|scales|factors|experiments|datasets|tasks|…`) as belt-and-suspenders.
+
+Corpus audit (101 PDFs, academic-normalized, old vs new): **5 papers changed, every change a correct restoration, zero regressions** — `plos_med_1` ("of 3 instruments"), `amp_1` ("sizes of 12 academic search engines"), `bmc_med_3` ("on 5 types of"), `ieee_access_5` ("kernel size to 3 x 3" + "for 30 iterations"), `maier_2023_collabra` ("Experiments 1a and 3 in Vastfall"). The fix repaired the filed defect **plus 4 pre-existing silent corruptions**. The 11 remaining R2 strips (all preceded by content words) are genuine page-number leaks, correctly still stripped.
+
+New regression tests in `tests/test_normalize_a3_r2_body_integer_real_pdf.py`: quantifier-head helper positives (`of 3 instruments`, `the 5 factors`), content-word-leak negative (`psychological 41 science` still strips), and a real-PDF assertion on `plos_med_1` (`of 3 instruments` present, `of instruments measuring` absent).
+
+> **Handoff items 1–3 (chen_2021_jesp) — won't-fix, PDF-text-layer corruption.** The same handoff filed three other defects on `chen_2021_jesp`: the lost "Registered reports: " title prefix (Nosek & Lakens), the "TurkPrime.com" → "TurkPrime. Com" space injection (Litman et al.), and the "Open Science Collaboration" → "Open, S. C." + stray "Psychology." mangle (OSC 2015). Verified that **both** MIT extractors — pdftotext (text channel) and pdfplumber (layout channel) — produce byte-identical corruption on all three. When two independent extractors read the same bytes, the corruption lives in the PDF's embedded text layer, not in any tool's reading order, and there is no in-text signal (the references read as clean and complete) to trigger a conditional fallback. No general structural signature exists to detect or repair them, and a tool swap is both forbidden (CLAUDE.md L-001 / AGPL ban) and would not help. These remain consumer-side / unrecoverable without OCR or AI re-read — the established `project_citationguard_extraction_defects_wontfix` class.
+
 ## [2.4.83] — 2026-06-08
 
 **P0r — bare "`<Initials> <Surname> et al.`" running-header strip.** `NORMALIZATION_VERSION` 1.9.29 → 1.9.30. Keyed on the `Initial. Surname et al.` line shape + the existing ≥3-standalone-repetition guard, never paper identity.

{docpluck-2.4.83 → docpluck-2.4.85}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.83
+Version: 2.4.85
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://docpluck.app
 Project-URL: Documentation, https://docpluck.app/api-docs

{docpluck-2.4.83 → docpluck-2.4.85}/TODO.md

@@ -146,6 +146,7 @@ Add only when a real downstream consumer asks for one. YAGNI until then.
 - [ ] **`### Reasons for change`** (ip_feldman) — Table 5 column header promoted to heading; needs table-region awareness (the body-coherence guard doesn't catch it because its body starts capitalized). RCA: rank-3 in the 2026-06-06 run-11 RCA.
 - [ ] **`## Data Availability` end-matter absent** — RCA CORRECTED the run-11 "demoter over-strip" premise: the section never enters the text channel (pdftotext drops the title-page box). Needs cross-channel (pdfplumber) recovery, same architecture class as B7. NOT a demoter exception.
 - [ ] **Glyph `Västfjäll`→`Vastfall`** (ar_apa/collabra, citationguard Defect 2) — baked pdftotext CID-font mis-map; needs a same-document surname-consensus normalizer (new subsystem). **Product/architecture decision on scope.**
+- [x] **Baked-glyph DIGIT misread `M_age 59.3`→`39.3`** (collabra.77859, surfaced 2026-06-08 RC-1 AI-verify) — **DIAGNOSED + DECISION MADE (2026-06-08): document as known limitation, no code change.** Same class as `Västfjäll` but a DIGIT in a statistic (silent stat corruption, the most dangerous form for meta-science): the PDF *visually* shows `59.3` but the embedded text codepoint is baked as `3`, and **both pdftotext AND pdfplumber faithfully extract `39.3`** (confirmed by visual PDF read + dual-extractor diff). No text-channel logic can recover it; the only fixes are OCR/multimodal-glyph-consensus (a new subsystem the user explicitly **declined** to scope this session). **Consumer note: CitationGuard / downstream stat-checkers must assume baked digit/letter misreads exist in source PDFs and apply their own cross-source (CrossRef/visual) verification — docpluck cannot guarantee a digit matches the visual glyph when the publisher baked the wrong codepoint.**
 - [ ] **org-author `Open Science Collaboration`→`Open, S. C.`** — baked into the PDF's embedded text by the publisher (identical in pdftotext AND pdfplumber); no safe general docpluck fix. **Routed to CitationGuard** (DOI/CrossRef author reconciliation).
 - [ ] **Tag v2.4.78** once the full canary set clears (currently 5 open). Then bump `PDFextractor/service/requirements.txt` pin + run `/docpluck-deploy`.
 
@@ -165,3 +166,17 @@ Add only when a real downstream consumer asks for one. YAGNI until then.
 - [ ] **Authenticated prod functional smoke** on chen via Railway /extract (needs a dp_ API key) — confirm O5 ordered-refs end-to-end in prod. Version (2.4.80) + health verified; the authenticated extract was not run.
 - [ ] **CitationGuard follow-ups** (their repo, documented in docs/DOCPLUCK_HANDOFF_2026-06-07.md): regenerate chen+jamison fixtures from docpluck v2.4.80 academic + re-score; extend citelink's number-ending-host special-case to single COVID-1928. Superscript recovery: WON'T-FIX in docpluck (would regress citelink — tested).
 - [ ] **ip_feldman interwoven table+prose case** (B4/#3/#4, R4/#5): still open; needs the per-y-band region-aware architecture (tracked in docs/superpowers/specs/2026-06-07-ip_feldman-B4-R4-column-interleave-diagnosis.md). O5 only handled the separable banded case (chen/jamison).
+
+## 2026-06-08 — v2.4.83 landed (footnote_texts API + extract_pdf_layout export + render label de-dup; committed fb0d595, PUSHED to origin/main, NOT tagged)
+
+> **Source:** ScienceArena arena-report verification session. The 2026-06-07 arena report (`ScienceArena/docs/reports/2026-06-07-docpluck-arena-issues.md`) blamed docpluck, but verification found the top findings (footnotes, body fidelity) were **ScienceArena adapter bugs**, not docpluck defects — full verdict in `ScienceArena/docs/reports/2026-06-08-docpluck-arena-issues-verified-response.md`. docpluck got 3 additive fixes (`report.footnote_texts`, top-level `extract_pdf_layout` export, render `### Table N` + `*Table N. …*` de-dup) + a load-aware fix to the DOCX/HTML perf tests (flaked the -n10 gate). Full suite **1906 passed**; the canary block on ip_feldman was a **proven false positive** (deterministic render diff: change touched only the 10 formal `### Table N` caption lines) — the 6 "new" findings were the case-normalization ledger bug (line ~165) re-flagging pre-existing B3/B4/column-interleave backlog → landed with `SKIP_CANARY=1` + proof in the commit message.
+
+### Release follow-ups (only when v2.4.83 is tagged)
+
+- [ ] **Bump `PDFextractor/service/requirements.txt`** git pin to `@v2.4.83` + update `PDFextractor/API.md` frozen-version examples. Production silently runs the old library until this lands (the `/docpluck-deploy` pre-flight check 4 enforces it).
+- [ ] **Run `/docpluck-deploy`** to ship 2.4.83 to prod after the pin bump + tag.
+- [ ] **Tag v2.4.83** (with the unreleased v2.4.81/82 on this RC branch) once the canary set clears. The strict no-exceptions canary gate fires on **tag** pushes, so the finding-key case-normalization bug (line ~165) must be fixed first or the tag canary will false-block.
+
+### ScienceArena adapter (their repo — fixed this session, verify there)
+
+- [ ] **Re-run the ScienceArena benchmark** with docpluck ≥ 2.4.83 installed in ITS venv. The adapter fixes (commit `de35f4a` on sciencearena `main`: pass `layout=`, read `report.footnote_texts`, strip the caption label) are logic-verified against local docpluck but NOT run end-to-end there (docpluck isn't installed in that repo). Also recommend ranking `docpluck-standard` as the primary real-document variant (Greek preserved).

{docpluck-2.4.83 → docpluck-2.4.85}/docpluck/__init__.py

@@ -78,7 +78,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.83"
+__version__ = "2.4.85"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.83 → docpluck-2.4.85}/docpluck/normalize.py

@@ -23,7 +23,7 @@ class NormalizationLevel(str, Enum):
     academic = "academic"
 
 
-NORMALIZATION_VERSION = "1.9.30"
+NORMALIZATION_VERSION = "1.9.32"
 
 
 # ── Mathematical Alphanumeric Symbols de-styling (shared, v2.4.34) ──────────
@@ -238,7 +238,59 @@ _REFS_END = re.compile(
 )
 _REF_START_VANCOUVER = re.compile(r"^\d{1,3}\.\s+[A-Z]")
 _REF_START_IEEE = re.compile(r"^\[\d+\]\s+[A-Z]")
-_REF_START_APA = re.compile(r"^[A-Z][a-z]+(?:-[A-Z][a-z]+)?,\s+[A-Z]\.")
+# Harvard / Cambridge name-year reference entry start (D1, citationguard-iterate
+# 2026-06-12): "Surname A and Surname B (2020) …", "Surname A et al. (2020) …",
+# "Surname A, Surname B and Surname C (2019) …". Distinct from APA, which puts a
+# comma immediately after surname-1 ("Surname, A.") — the Harvard form has NO
+# comma between surname and initials, so _REF_START_APA never matched it and R3
+# collapsed the whole bibliography onto one line (British Journal of Political
+# Science bjps_1: 109 entries joined into a single paragraph). The structural
+# signature is an author block — Surname + 1–3 bare initials, optionally chained
+# with " and "/"&"/comma — terminated by a parenthesised 4-digit year. The
+# parenthesised year is the strong anchor that keeps mid-entry wrap lines
+# ("American Journal of\nPolitical Science 64, 904-20.") from matching.
+# Initials: 1-4 groups, each 1-3 capitals, optionally hyphenated ("H-G", "Z-C")
+# and optionally period-terminated; groups may be spaced ("R J") or glued ("DH").
+_HARVARD_INITIALS = r"(?:\s+[A-Z]{1,3}(?:-[A-Z]{1,3})?\.?){1,4}"
+# Surname: Title-case word (Latin-Extended so "Häusermann", "Öhman" qualify),
+# optionally hyphenated ("Huntington-Klein") and optionally a second word for
+# compound surnames ("Santos Silva", "El Soufi"), with leading lowercase
+# particles ("van der", "de la") permitted.
+# Letter ranges span ASCII + Latin-1 Supplement + Latin Extended-A/B (U+0100-
+# U+024F) so Eastern-European / Turkish surnames ("Häusermann", "Tuğal", "Öhman")
+# qualify. First letter is an uppercase-ish Latin letter; the rest may be any
+# Latin letter or apostrophe.
+_HARVARD_NAME_WORD = (
+    r"[A-ZÀ-ÞĀ-ɏ][A-Za-zÀ-ÿĀ-ɏ'’]+(?:-[A-ZÀ-ÞĀ-ɏ][A-Za-zÀ-ÿĀ-ɏ'’]+)?"
+)
+_HARVARD_SURNAME = (
+    # Leading particles, capitalised or not ("van der", "de la", "Van der Brug",
+    # "De Vries") — the first particle is often title-cased at an entry start.
+    r"(?:(?:[Vv]an|[Vv]on|[Dd]e|[Dd]er|[Dd]en|[Dd]i|[Dd]el|[Dd]ella|[Dd]u"
+    r"|[Ll]a|[Ll]e|[Ee]l|[Dd]os|[Dd]a)\s+){0,2}"
+    + _HARVARD_NAME_WORD
+    + r"(?:\s+" + _HARVARD_NAME_WORD + r")?"
+)
+_HARVARD_AUTHOR = _HARVARD_SURNAME + _HARVARD_INITIALS
+_REF_START_HARVARD = re.compile(
+    r"^" + _HARVARD_AUTHOR +
+    r"(?:"
+    r",?\s+et\s+al\.?"                          #   "et al." / "Surname K, et al."
+    r"|,?\s+(?:and|&)\s+" + _HARVARD_AUTHOR +   # " and Surname I"
+    r"|,\s+" + _HARVARD_AUTHOR +                # ", Surname I"
+    r")*"
+    r"(?:\s+\((?:eds?|editors?)\.?\))?"         # optional "(eds)" / "(ed.)" marker
+    r"\s+\((?:1[89]|20)\d{2}[a-z]?\)",         # (Year) optional letter suffix
+    re.UNICODE,
+)
+# APA reference entry start: "Surname, A.", "Surname, A. B.". Reuses the shared
+# surname block so accented ("Yücel, M."), particle ("de Kovel, C."), and
+# compound ("Karlsson Linnér, R.") surnames are recognised — the previous
+# ASCII-only `[A-Z][a-z]+` form silently merged those entries into the preceding
+# reference (surfaced on nat_comms_5 / nathumbeh_2 during the D1 broad-read;
+# same root-cause class as D1). The comma-then-initial is the APA discriminator
+# (Harvard has no comma after the surname).
+_REF_START_APA = re.compile(r"^" + _HARVARD_SURNAME + r",\s+[A-Z]\.", re.UNICODE)
 
 
 def _find_references_spans(text: str) -> list[tuple[int, int]]:
@@ -258,7 +310,17 @@ def _find_references_spans(text: str) -> list[tuple[int, int]]:
         ref_starts = (
             len(re.findall(r"\b\d{1,3}\.\s+[A-Z]", window))
             + len(re.findall(r"\n\[\d+\]\s+[A-Z]", window))
-            + len(re.findall(r"\n[A-Z][a-z]+(?:-[A-Z][a-z]+)?,\s+[A-Z]\.", window))
+            + len(re.findall(r"\n" + _HARVARD_SURNAME + r",\s+[A-Z]\.", window))
+            # Harvard name-year entries (D1): "\nSurname A(?: and …| et al.)? (YYYY)".
+            # Without this a pure-Harvard bibliography (no comma after surname,
+            # no numbered/IEEE entries) would score 0 ref-starts and the span
+            # would go undetected, so R3 would never run to keep entries split.
+            + len(re.findall(
+                r"\n" + _HARVARD_AUTHOR +
+                r"(?:\s+et\s+al\.?|,?\s+(?:and|&)\s+" + _HARVARD_AUTHOR +
+                r")*\s+\((?:1[89]|20)\d{2}[a-z]?\)",
+                window,
+            ))
         )
         if ref_starts >= 3:
             end_m = _REFS_END.search(text, start)
@@ -469,23 +531,94 @@ _R2_BODY_NOUN_PATTERN = re.compile(
     r"people|persons?|adults?|children|students?|patients?|workers?|"
     r"employees?|managers?|leaders?|followers?|users?|members?|"
     r"votes?|comments?|ratings?|reviews?|posts?|tweets?|messages?|"
-    r"items?|conditions?|variables?|categories?|topics?|themes?)\b",
+    r"items?|conditions?|variables?|categories?|topics?|themes?|"
+    r"instruments?|measures?|scales?|factors?|dimensions?|domains?|"
+    r"experiments?|datasets?|samples?|tasks?|stimuli|questions?)\b",
     re.IGNORECASE,
 )
 
+# v2.4.84 (NORMALIZATION_VERSION 1.9.31): R2 quantifier-head pre-context guard.
+#
+# The body-noun allowlist above is necessarily incomplete — it can never
+# enumerate every countable noun a reference title might quantify ("3
+# instruments", "5 trajectories", "12 heuristics", …). The amle_1 fix
+# (v2.4.17) added nouns one at a time; the plos_med_1 "Clinimetric properties
+# of 3 instruments" → "… of instruments" drop (filed by citationguard-iterate
+# 2026-06-10, same class as the earlier Mayiwar case) is the same whack-a-mole
+# recurring.
+#
+# A genuine page-number leak and a legitimate quantifier are distinguished by
+# the word IMMEDIATELY PRECEDING the digit, not the noun after it:
+#   * quantifier — the digit heads a noun phrase, so it follows a CLOSED-CLASS
+#     function word (article / preposition / determiner): "of 3 instruments",
+#     "the 5 factors", "first 20 years", "only 3 studies".
+#   * page leak — the digit interrupts a content phrase, so it follows a
+#     CONTENT word (adjective / noun): "psychological 41 science",
+#     "recovery 12 in a population".
+# Function words are a finite closed class, so keying on them generalizes where
+# the open-ended noun list cannot. This guard is purely ADDITIVE — it only
+# ever PRESERVES a digit (returns True), never strips one — so it cannot make
+# R2 strip anything it did not already strip (no false-positive page numbers
+# newly retained beyond the safe direction). Per docpluck's correctness
+# asymmetry, silently deleting a digit from a scientific reference title (rule
+# 0a, NO TEXT MAY DISAPPEAR) is far worse than leaving a stray page number, so
+# biasing toward preserve at a quantifier head is the correct trade.
+_R2_QUANTIFIER_HEAD_WORDS = frozenset(
+    {
+        # articles
+        "a", "an", "the",
+        # prepositions
+        "of", "in", "on", "at", "to", "for", "with", "by", "from", "as",
+        "into", "than", "between", "among", "amongst", "through", "during",
+        "after", "before", "over", "under", "about", "across", "per",
+        "within", "upon", "against", "toward", "towards", "around",
+        # conjunctions
+        "and", "or", "nor", "but",
+        # determiners / quantifier-context heads
+        "all", "both", "these", "those", "some", "any", "each", "every",
+        "first", "last", "next", "only", "total", "following", "remaining",
+        "top", "approximately", "nearly", "least", "most", "up", "least",
+        "another", "additional", "further", "respective", "successive",
+    }
+)
+# Trailing-word extractor: the alphabetic word ending right before ``match_pos``
+# (the optional trailing hyphen lets "well-3" style hyphenations still resolve
+# to the final segment, which is what matters for the function-word lookup).
+_R2_PRECEDING_WORD = re.compile(r"([A-Za-z]+)[\s ]*$")
+
 
 def _r2_is_body_phrase(digit_str: str, refs_text: str, match_pos: int) -> bool:
     """Return True if the digit at ``match_pos`` is part of a body phrase
-    (e.g. "20 years", "1,675 participants") and should NOT be stripped by R2.
-
-    Heuristic: check the 30-char window AFTER the matched digit for a
-    body-noun keyword (years, participants, etc.). If found, the digit is
-    almost certainly part of legitimate body prose, not a page-number leak.
+    (e.g. "20 years", "1,675 participants", "of 3 instruments") and should
+    NOT be stripped by R2 (the page-number scrub).
+
+    Two complementary, both-conservative signals — either one preserves:
+
+    1. **Following body-noun** — the 60-char window AFTER the digit contains a
+       known body-noun keyword (years, participants, instruments, …). Handles
+       "20 years", "1,675 participants".
+    2. **Preceding quantifier head** — the word IMMEDIATELY BEFORE the digit is
+       a closed-class function word (article / preposition / determiner), so
+       the digit heads a quantified noun phrase ("of 3 instruments", "the 5
+       factors") rather than interrupting a content phrase. Generalizes beyond
+       the finite noun list. See ``_R2_QUANTIFIER_HEAD_WORDS``.
+
+    A genuine page-number leak ("psychological 41 science", "recovery 12 in a
+    population") fails BOTH checks — the noun after isn't in the list and the
+    word before is a content word — so it is still stripped.
     """
-    # Window starts after the digit + at least one space.
+    # (1) Following body-noun. Window starts after the digit + at least one space.
     window_start = match_pos + len(digit_str)
     window = refs_text[window_start:window_start + 60]
-    return bool(_R2_BODY_NOUN_PATTERN.search(window))
+    if _R2_BODY_NOUN_PATTERN.search(window):
+        return True
+
+    # (2) Preceding quantifier head (closed-class function word).
+    m = _R2_PRECEDING_WORD.search(refs_text[:match_pos])
+    if m and m.group(1).lower() in _R2_QUANTIFIER_HEAD_WORDS:
+        return True
+
+    return False
 
 
 def _looks_like_ref_start(line: str) -> bool:
@@ -493,6 +626,7 @@ def _looks_like_ref_start(line: str) -> bool:
         _REF_START_VANCOUVER.match(line)
         or _REF_START_IEEE.match(line)
         or _REF_START_APA.match(line)
+        or _REF_START_HARVARD.match(line)
     )
 
 
@@ -1458,6 +1592,32 @@ _AUTHOR_ETAL_INITIAL = re.compile(
 )
 
 
+# D2 (citationguard-iterate 2026-06-12): single-word / short category-label
+# running headers. Nature-family and many journals print the article-type label
+# ("Article", "Review", "Letter", "Matters Arising", …) at the top of every page.
+# H0 already curates these in _HEADER_BANNER_PATTERNS, but H0 only fires in the
+# document-header zone (first 30 lines); when the label recurs mid-document — e.g.
+# inside the References section at a page break — it survives and gets welded into
+# an entry ("…EAE based on\n\n\x0cArticle histology…" orphaned ref 34's year on
+# nat_comms_2). Stripping it here is gated by the ≥3-standalone-repetition guard in
+# _detect_recurring_running_headers, so a one-off body occurrence is never touched.
+# Scoped to genuine publisher article-type furniture; bare common words ("Research",
+# "Comment") are excluded to avoid colliding with section-heading body lines.
+_CATEGORY_LABEL_HEADER = re.compile(
+    r"^(?:"
+    r"Article|ARTICLE|Articles"
+    r"|Review|Reviews|REVIEW"
+    r"|Letter|Letters|LETTER"
+    r"|Resource|Resources"
+    r"|Analysis|Perspective|Perspectives"
+    r"|Correspondence|Editorial"
+    r"|Brief\s+Communication|Matters\s+Arising"
+    r"|Original\s+(?:Investigation|Article|Research)"
+    r"|Research\s+(?:Article|Paper|Letter|Report)"
+    r")$"
+)
+
+
 def _is_all_caps_journal_banner(line: str) -> bool:
     """All-caps multi-word journal banner: e.g. ``PLOS MEDICINE``,
     ``COGNITION AND EMOTION``, ``JAMA NETWORK OPEN``.
@@ -1501,6 +1661,8 @@ def _looks_like_running_header_or_footer(line: str) -> bool:
     """
     if not line or len(line) > 100:
         return False
+    if _CATEGORY_LABEL_HEADER.match(line):
+        return True
     if _is_all_caps_journal_banner(line):
         return True
     if _AUTHOR_PAIR_ALL_CAPS_AND.match(line):
@@ -3555,6 +3717,22 @@ def normalize_text(
         for r_start, r_end in reversed(_refs_spans):
             refs_text = t[r_start:r_end]
 
+            # R3 page-break stitch (D2, citationguard-iterate 2026-06-12): inside
+            # a bibliography a form-feed (page break) NEVER coincides with a
+            # paragraph boundary — entries are delimited by ref-starts, not blank
+            # lines. When an entry straddles a page break, pdftotext emits
+            # "…based on\n\n\x0cArticle\nhistology…(2008)." — the blank line +
+            # form feed split the entry, so R3's normal continuation join (which
+            # resets on a blank line) leaves the tail (and its year) detached,
+            # orphaning nat_comms_2 ref 34's "(2008)". Collapse each form-feed
+            # junction (and the blank line(s) around it) to a single newline so
+            # the tail rejoins the head as an ordinary continuation. The running-
+            # header label on the new page ("Article") is stripped upstream by
+            # P0r (_CATEGORY_LABEL_HEADER); any that survives is handled by the
+            # continuation join, but the year is recovered regardless.
+            refs_text = re.sub(r"[ \t]*\n[ \t\n]*\f[ \t]*", "\n", refs_text)
+            refs_text = refs_text.replace("\f", "\n")
+
             # R3 pre-pass (Cycle 15 v2.4.67): two-column bibliography
             # pairing. pdftotext renders some 2-column bibliographies by
             # streaming the entire NUMBER column first, then the entire
@@ -3582,15 +3760,49 @@ def normalize_text(
             before_r3 = refs_text
             lines = refs_text.split("\n")
             joined: list[str] = []
+            # Index in ``joined`` of the entry currently being built. Unlike the
+            # previous ``joined[-1]`` check, this survives blank lines: within a
+            # bibliography a blank line is a page-break artifact (entries are
+            # single-newline separated), so a continuation line that follows a
+            # blank still belongs to the entry above it. This rejoins entries
+            # split across a page break — e.g. nat_comms_2 ref 34, whose
+            # "(2008)" year sat past the page-break blank after the running
+            # header was stripped (D2). Entry boundaries are still established
+            # solely by _looks_like_ref_start, so genuinely separate entries do
+            # not merge.
+            cur_entry = -1
+            saw_blank = False
             for line in lines:
                 stripped = line.strip()
                 if not stripped:
                     joined.append("")
+                    saw_blank = True
                     continue
-                if joined and joined[-1] and not _looks_like_ref_start(stripped):
-                    joined[-1] = joined[-1].rstrip() + " " + stripped
-                else:
+                is_start = _looks_like_ref_start(stripped)
+                bridged = False
+                if cur_entry >= 0 and not is_start:
+                    if saw_blank:
+                        # Crossing a blank line: only bridge when the current
+                        # entry is syntactically INCOMPLETE (does not end with
+                        # sentence-terminal punctuation). A page-break split
+                        # leaves the head mid-clause ("…EAE based on"), so the
+                        # tail rejoins; a COMPLETED entry ("…46, 215-39.")
+                        # followed by a blank is the end of the list, and the
+                        # next block ("Cite this article: …") is post-reference
+                        # trailer that must NOT be absorbed.
+                        prev = joined[cur_entry].rstrip()
+                        if prev and prev[-1] not in ".?!":
+                            joined[cur_entry] = prev + " " + stripped
+                            bridged = True
+                    else:
+                        joined[cur_entry] = (
+                            joined[cur_entry].rstrip() + " " + stripped
+                        )
+                        bridged = True
+                if not bridged:
                     joined.append(stripped)
+                    cur_entry = len(joined) - 1
+                saw_blank = False
             refs_text = "\n".join(joined)
             r3_joins_total += before_r3.count("\n") - refs_text.count("\n")
 

{docpluck-2.4.83 → docpluck-2.4.85}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docpluck"
-version = "2.4.83"
+version = "2.4.85"
 description = "PDF, DOCX, and HTML text extraction and normalization for academic papers"
 readme = "docs/README.md"
 requires-python = ">=3.10"