docpluck 2.4.45__tar.gz → 2.4.46__tar.gz

{docpluck-2.4.45 → docpluck-2.4.46}/.claude/skills/docpluck-iterate/references/ai-full-doc-verify.md

@@ -68,6 +68,8 @@ generate-gold <absolute-path-to-PDF>
 
 `article-finder`'s `generate-gold` runs `gold-generation.md` end to end — the canonical extraction prompt(s), anti-hallucination rules, cross-check, schema validation, and `register-view` storage under the canonical key. It produces (and registers) the `reading`, `citations`, and `stats` views in one pass. This is the ONLY sanctioned way for docpluck-iterate to obtain a gold that does not already exist.
 
+> **Prerequisite — the `codex` CLI must be installed AND authenticated.** `gold-generation.md` runs an independent Codex / GPT-5.5 cross-model audit of every gold before storage, and `generate-gold` **blocks** (rather than shipping unverified gold) if `codex` is missing or unauthenticated. Verify ahead of time with `codex --version`. If a `generate-gold` call fails on the cross-model audit step, the user must run `codex login` — it is interactive, so the skill cannot do it; surface this to the user as a blocker rather than falling back to a local extraction.
+
 After `generate-gold` completes, the `reading` view is in the shared cache — copy it to the working path exactly as in the cache-HIT branch:
 
 ```bash

{docpluck-2.4.45 → docpluck-2.4.46}/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## [2.4.46] — 2026-05-16
+
+**Cycle G5c-1 (APA-first run) — orphan multi-level section number stranded above its heading (G5c, S1).** pdftotext sometimes splits a numbered subsection heading such as `5.4. Discussion` into a bare `5.4.` line and a separate `Discussion` line; the section partitioner then promotes the lone title word to a generic `## Discussion` and strands the number on its own line. In `jdm_m.2022.2` the `5.4. Discussion` subsection of Study 1 rendered as an orphan `5.4.` followed by a top-level `## Discussion`.
+
+Fix (v2.4.46) — new render post-processor `_fold_orphan_multilevel_numerals_into_headings`, the multi-level analogue of `_fold_orphan_arabic_numerals_into_headings` / `_fold_orphan_roman_numerals_into_headings`. It folds an orphan `N.N.` number into the **immediately-adjacent** generic `##`/`###` heading and emits it at subsection level: `5.4.`⏎`## Discussion` → `### 5.4. Discussion`. Keyed purely on the structural signature (an isolated multi-level dotted number is itself a strong subsection marker — body prose and list items never emit a bare `5.4.` line) plus blank-line-only adjacency. `### Figure N` / `### Table N` (library-emitted structural markers) and already-numbered headings are excluded. Only the immediately-adjacent case is folded; an orphan number whose title word the partitioner consumed elsewhere (leaving body prose below the number) is partitioner-level work (G5c-2) and is left untouched.
+
+`jdm_m.2022.2`: the `5.4. Discussion` heading is recovered and AI-gold-verified correct. The v2.4.45→v2.4.46 diff is heading-markup only (0 text loss, 0 hallucination). 26/26 baseline PASS. New real-PDF + contract tests in `tests/test_orphan_multilevel_number_real_pdf.py`.
+
+~11 APA papers still FAIL Phase-5d verification (G5c-2 partitioner split-heading rejoin, HALLUC-HEAD, FIG caption double-emission, TABLE cluster, COL); the run continues.
+
 ## [2.4.45] — 2026-05-16
 
 **Cycle 13 (autonomous APA-first run) — long descriptive numbered headings demoted to body text (G5b, S1).** `render.py`'s numbered-heading promoters carried a "long lowercase-word run" prose guard (`max_lc_run >= 5`) that rejected legitimate descriptive headings — e.g. `2.4.2.2. Inference of planning strategies and strategy types`, `3.3.2.1. The quality of planning on the previous trial moderates the effect of reflection`. jdm_.2023.16 alone had 19 multi-level numbered subsection headings demoted to body text.

{docpluck-2.4.45 → docpluck-2.4.46}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: docpluck
-Version: 2.4.45
+Version: 2.4.46
 Summary: PDF, DOCX, and HTML text extraction and normalization for academic papers
 Project-URL: Homepage, https://github.com/giladfeldman/docpluck
 Project-URL: Documentation, https://github.com/giladfeldman/docpluck/tree/main/docs

{docpluck-2.4.45 → docpluck-2.4.46}/docpluck/__init__.py

@@ -71,7 +71,7 @@ from .figures import Figure
 from .extract_structured import TABLE_EXTRACTION_VERSION, StructuredResult, extract_pdf_structured
 from .render import render_pdf_to_markdown
 
-__version__ = "2.4.45"
+__version__ = "2.4.46"
 __author__ = "Gilad Feldman"
 __license__ = "MIT"
 

{docpluck-2.4.45 → docpluck-2.4.46}/docpluck/render.py

@@ -772,6 +772,48 @@ def _fold_orphan_arabic_numerals_into_headings(text: str) -> str:
     return pattern.sub(repl, text)
 
 
+def _fold_orphan_multilevel_numerals_into_headings(text: str) -> str:
+    """Cycle G5c-1: fold an orphan multi-level section-number line (``5.4.``,
+    ``6.1.2.``) into the immediately following generic ``##``/``###`` heading.
+
+    Multi-level analogue of :func:`_fold_orphan_arabic_numerals_into_headings`.
+    pdftotext sometimes splits ``5.4. Discussion`` into a bare ``5.4.`` line
+    and a separate ``Discussion`` line; the section partitioner then promotes
+    the lone title word to a generic ``## Discussion`` and strands the number::
+
+        5.4.\\n\\n## Discussion  →  ### 5.4. Discussion
+
+    A multi-level dotted number alone on a line is itself a strong subsection
+    signal — body prose and list items do not emit a bare ``5.4.`` line — so
+    the fold is keyed purely on that structural signature plus blank-line-only
+    adjacency to a heading. The result is always ``### ``: multi-level
+    numbering denotes a subsection regardless of the level the partitioner
+    happened to give the stranded title (cf. ``_NUMBERED_SUBSECTION_HEADING_RE``,
+    which likewise emits ``### `` at any depth).
+
+    The fold target must be a *generic* heading. ``### Figure N`` / ``### Table N``
+    are library-emitted structural markers, and a heading already starting with
+    a number is a real numbered section — both are excluded (the latter also
+    keeps the pass idempotent). Only the immediately-adjacent case is folded;
+    an orphan number separated from its heading by a figure block or by body
+    prose (the title word consumed elsewhere) is partitioner-level work and is
+    left untouched here.
+    """
+    if not text:
+        return text
+    pattern = re.compile(
+        r"(?m)^(\d+(?:\.\d+){1,3})\.?[ \t]*\n(?:[ \t]*\n)+"
+        r"(?P<head>#{2,3} (?!\s*\d)(?!Figure\b)(?!Table\b)[^\n]+)"
+    )
+
+    def repl(m: re.Match) -> str:
+        num = m.group(1)
+        head_text = m.group("head").split(" ", 1)[1]
+        return f"### {num}. {head_text}"
+
+    return pattern.sub(repl, text)
+
+
 def _promote_study_subsection_headings(text: str) -> str:
     """Promote ``Study N Design and Findings`` etc. to ``### {title}``.
 
@@ -2159,6 +2201,10 @@ def render_pdf_to_markdown(
     # heading post-processors so it operates on the final heading shapes.
     md = _fold_orphan_roman_numerals_into_headings(md)
     md = _fold_orphan_arabic_numerals_into_headings(md)
+    # Cycle G5c-1: multi-level analogue — fold an orphan `N.N.` number line
+    # into the immediately following generic heading (`5.4.`\n\n`## Discussion`
+    # -> `### 5.4. Discussion`). Runs alongside the single-level folders.
+    md = _fold_orphan_multilevel_numerals_into_headings(md)
     # Cycle 11 (G5a): promote single-level `N. Title` lines to `## N. Title`,
     # gated on the document already numbering its sections. Runs AFTER the
     # orphan-numeral folders so `## 1. Introduction` exists as an anchor.

{docpluck-2.4.45 → docpluck-2.4.46}/docs/HANDOFF_2026-05-16_ai-gold-instructions.md

@@ -30,12 +30,10 @@ docpluck must change three things:
   now rich enough for docpluck's TABLE verifier.
 - `register-view` and `migrate` now **reject a non-canonical key** with an
   actionable error.
-- `gold-generation.md` now enforces a **100%-accuracy, zero-hallucination policy**
-  and an **independent Codex / GPT-5.5 cross-model verification**: a second-vendor
-  model re-reads the PDF and audits every gold before it is stored. **Ensure the
-  `codex` CLI is installed and authenticated** in docpluck's environment
-  (`codex --version`; `codex login` if needed) — `generate-gold` blocks without it
-  rather than shipping unverified gold.
+- `gold-generation.md` enforces a **100%-accuracy, zero-hallucination policy**:
+  dual independent extraction + cross-check, verbatim anchors, schema validation.
+  (An independent cross-vendor verification pass is planned but not yet live — see
+  article-finder's `TODO.md`.)
 
 ---
 

docpluck-2.4.46/docs/HANDOFF_2026-05-16_iterate_run_4_final.md

@@ -0,0 +1,109 @@
+# Handoff — docpluck-iterate run 4 (fix-and-continue) — FINAL
+
+**Authored:** 2026-05-16, end of run 4. **For:** a fresh `/docpluck-iterate` session.
+This run executed `docs/HANDOFF_2026-05-16_iterate_run_4_fix_and_continue.md`'s three jobs.
+
+## State at handoff
+
+- Last shipped library version: **v2.4.45** (tag pushed, PyPI not published).
+- docpluckapp `service/requirements.txt` pin: auto-bumped to **v2.4.45** (commit `26bf88f9`).
+- Production `/_diag`: `docpluck_version = 2.4.45` — verified.
+- 26-paper baseline: **26/26 PASS** at v2.4.45.
+- Broad pytest: **0 failures** (the 15 pre-existing failures are resolved — see tests-regen).
+
+## What run 4 shipped
+
+| Item | Version | Outcome |
+|---|---|---|
+| JOB 1 — cycle 12 ligature rework | v2.4.44 | SHIPPED + prod-verified |
+| JOB 3 — tests-regen | (no bump) | SHIPPED (`c831e28`) |
+| JOB 3 — cycle 13 (G5b) | v2.4.45 | SHIPPED + prod-verified |
+| JOB 2 — 3 fragmented golds | (cache) | REGISTERED under canonical DOI keys |
+| skill — codex prerequisite note | (no bump) | committed `9aa4f5b` |
+
+### JOB 1 — cycle 12 ligature rework (v2.4.44)
+The session-3 cycle-12 attempt was broken (a duplicate `decompose_ligatures` call
+before the pre-existing S3 step starved S3's tracking; `test_report_tracks_changes`
+red). Reworked: removed the duplicate, unified S3 to call the single shared helper
+(explicit U+FB00-FB06 ASCII table — NFKC of `ſt` yields a non-ASCII long-s), kept the
+genuine `cell_cleaning` + render-post-process calls (table/caption/fence channels
+bypass `normalize_text`). Stale narrative in CHANGELOG/LEARNINGS/lessons/TRIAGE
+corrected. v2.4.44→ diff is ligature-only on korbmacher/jdm_m2/jdm16; 26/26.
+
+### JOB 3 — tests-regen (`c831e28`, no version bump)
+12 `test_extract_pdf_byte_identical` snapshots + 2 `test_sections_golden` goldens
+regenerated (environmental pdftotext line-wrap drift; `extract_pdf` is a pure
+pdftotext passthrough). The 15th failure, `test_request_09`, is **NOT** snapshot
+drift — it is a real **COL-class** column-interleave defect (the numbered RSOS
+bibliography renders as `References\n1. 2. 3. … 16.\n\nThaler RH…`, the number column
+split from the entry text). Left red; tracked as the COL class.
+
+### JOB 3 — cycle 13 (G5b, v2.4.45)
+`render.py`'s numbered-heading promoters carried a `max_lc_run >= 5` prose guard
+that demoted long descriptive headings. Reproduction showed real headings with
+lowercase-runs up to 12 — the count cannot separate heading from prose, so "raise
+5→8" would have been a partial fix. Removed the guard from
+`_promote_numbered_subsection_headings` (multi-level dotted numbering is itself the
+discriminator); kept it raised 5→8 in `_promote_numbered_section_headings`
+(single-level numbers collide with enumerated lists). jdm_.2023.16: 19 headings
+recovered; v2.4.44→v2.4.45 diff heading-promotion-only; 26/26.
+
+### JOB 2 — 3 fragmented golds (Chen / Xiao / Efendic)
+Regenerated all four views (`stats` / `reading` / `citations` / `intext_citations`)
+for each of the 3 papers through `gold-generation.md` — dual stats extraction +
+cross-check + reading/citations/intext carrier pass (12 subagents). Registered all
+12 views under canonical DOI keys (`10.1016__j.jesp.2021.104154`,
+`10.1080__23743603.2021.1878340`, `10.1177__19485506211056761`), producer
+`article-finder`; `ai-gold.py audit` clean.
+
+**Codex Step-4 cross-model verification was SKIPPED** (explicit user directive,
+2026-05-16) because the `codex` CLI has a Windows UTF-8 file-read bug — it misreads
+UTF-8 gold files as mojibake, flooding the verdict with false discrepancies. A full
+report is at `~/ArticleRepository/docs/handoffs/2026-05-16_codex-cli-windows-encoding-issue.md`
+(handed to the article-finder skill owner). The UTF-8-corrected Codex re-runs still
+found **genuine** gold discrepancies (chen 28 / xiao 22 / efendic 19 — real citation
+page-range swaps, missing title prefixes, a few wrong table cells) — the verdict
+files are saved at `~/ArticleRepository/tmp/goldgen_run4/{chen,xiao,efendic}_verdict2.txt`
+for article-finder's Step-4 fix-loop. The regenerated golds are dual-extracted +
+cross-checked and supersede the old docpluck private-prompt golds, but they have NOT
+passed Codex — a fix-loop is still owed (now article-finder's, per the directive).
+
+## Open queue — JOB 3 remaining APA defect cycles (the run did NOT finish JOB 3)
+
+**Standing verdict (rule 0e-bis): the APA corpus is NOT clean — ~11 papers still
+FAIL Phase-5d.** This run shipped cycle 13 and re-scoped G5c; the cycles below remain.
+Recommended order:
+
+1. **G5c-1** — render-layer fold of an orphan multi-level `N.N.` line into an
+   adjacent generic `##/###` heading (the `5.4.`/`## Discussion` case). C1-C2,
+   ships independently. See TRIAGE "Cycle 14 (investigation)".
+2. **FIG caption double-emission + truncation** — ~8 papers. S2, C2.
+3. **G5c-2 + G5d + TABLE** — the section-partitioner cluster, C3, a dedicated
+   session: G5c-2 (split-heading rejoin — pdftotext splits `N.N. Title` and the
+   partitioner consumes the title word; 5 of 6 jdm_m2 cases), G5d (named/unnumbered
+   heading demotion, ~7 papers), TABLE structure destruction (~11 papers, the single
+   largest blocker).
+4. **COL column-interleave** (incl. `test_request_09`'s numbered-bibliography split)
+   and **GLYPH** deleted-minus — S0, C3-C4, layout-channel; escalate.
+
+`test_request_09` will stay red until the COL class is fixed — it is a correct
+regression test catching a real defect, not a stale fixture.
+
+## Process notes / improvements
+
+- The `codex` CLI Windows UTF-8 bug (above) is article-finder's `gold-generation.md`
+  Step-4 to fix — report filed in the ArticleRepository handoffs dir.
+- `ai-full-doc-verify.md` Step 1c now states the codex prerequisite (committed).
+- Cross-skill lesson re-confirmed twice this run
+  (`reproduce-triage-defect-at-head-before-trusting-cost-estimate`): both G5b and
+  G5c were costed wrong in the TRIAGE — G5b deeper (guard removal, not 5→8), G5c
+  deeper (partitioner, not a render fold). Always reproduce + measure at HEAD.
+
+## Stop reason
+
+Run 4 completed JOB 1, JOB 2, and 2 of JOB 3's items (tests-regen + cycle 13), and
+re-scoped G5c. Stopped before the remaining JOB 3 cycles because they are a
+fresh-session-sized block of C2-C3 section-partitioner work (G5c-2 / G5d / TABLE)
+plus escalated C3-C4 layout-channel work (COL / GLYPH) — continuing to grind them in
+an already-very-long session is the wrong call. The next `/docpluck-iterate` session
+resumes at G5c-1 from the queue above.

docpluck-2.4.46/docs/HANDOFF_2026-05-16_iterate_run_4_fix_and_continue.md

@@ -0,0 +1,157 @@
+# Handoff — docpluck-iterate run 4: fix-and-continue (fresh session)
+
+**Authored:** 2026-05-16, end of session 3. **For:** a fresh `/docpluck-iterate` session.
+**Read this whole file before touching anything.** It is self-contained — it assumes no memory of session 3.
+
+You have **three jobs, in this order**:
+
+1. **JOB 1 — Resolve the in-flight cycle 12 (ligature fix).** It is committed-nowhere, sitting uncommitted in the working tree, and it is **broken** — it introduced a test regression and duplicates an existing normalize step. Decide: rework it, or revert it. **Do not commit it as-is.**
+2. **JOB 2 — Finish the article-finder AI-gold integration.** ArticleFinder shipped its side and left docpluck a punch-list (`docs/handoffs/` — see §JOB 2). docpluck's session-3 fix (commit `ac34c7e`) did part of it; the rest is open.
+3. **JOB 3 — Continue the APA iteration loop** from the TRIAGE punch-list.
+
+Invoke the `docpluck-iterate` skill normally (it runs its own preflight). Then work these three jobs.
+
+---
+
+## 0. Immediate git / working-tree state
+
+- Repo: `C:\Users\filin\Dropbox\Vibe\MetaScienceTools\docpluck`, branch `main`.
+- Last commit: `5cc321a docs: add AI-gold instructions from article-finder coordination`.
+- Recent history (all session 3, all clean, all prod-deployed): `bbad28f` v2.4.43 (cycle 11), `ac34c7e` skill-fix (gold delegation), `9b41e4d` v2.4.42 (cycle 10), `951b00a` v2.4.41-ish… — i.e. **v2.4.43 is the last shipped library version.**
+- **Uncommitted working tree = the cycle-12 ligature attempt (v2.4.44).** Modified: `docpluck/normalize.py`, `docpluck/render.py`, `docpluck/tables/cell_cleaning.py`, `docpluck/__init__.py`, `pyproject.toml`, `CHANGELOG.md`, `docs/TRIAGE_2026-05-14_phase_5d_gold_audit.md`, `.claude/skills/docpluck-iterate/LEARNINGS.md`, `.claude/skills/_project/lessons.md`; untracked new file `tests/test_ligature_decomposition_real_pdf.py`.
+- 26-paper baseline at the cycle-12 working tree: **26/26 PASS, 0 WARN** (re-confirmed).
+- Broad pytest at the cycle-12 working tree: **16 failed / 1233 passed**. 15 of the 16 are the long-standing pre-existing set (12× `test_extract_pdf_byte_identical` snapshot drift + 2× `test_sections_golden` + 1× `test_request_09`). **The 16th is cycle-12-introduced** — see JOB 1.
+
+---
+
+## JOB 1 — Cycle 12 (ligature decomposition) is BROKEN. Rework or revert.
+
+### What cycle 12 attempted
+
+Goal: decompose Latin typographic ligatures (`ff fi fl ffi ffl ſt st`, U+FB00-FB06) — a corpus scan found them in 35 rendered `.md` files (`korbmacher` 82×, `jdm_.2023.16` 34×). The attempt added a new `normalize.py::decompose_ligatures` helper (per-char NFKC scoped to `[ff-st]`) and wired it into three channels: `normalize_text` body (right after the NFC step, ~line 1567), `tables/cell_cleaning._html_escape`, and `render_pdf_to_markdown` post-process. Bumped to v2.4.44, NORMALIZATION_VERSION 1.9.8.
+
+### Why it is broken
+
+**docpluck `normalize.py` ALREADY HAS a ligature-expansion step** — `S3_ligature_expansion` at **`normalize.py` ~line 1687**:
+
+```python
+t = t.replace("ff", "ff")   # ff
+t = t.replace("fi", "fi")   # fi
+t = t.replace("fl", "fl")   # fl
+t = t.replace("ffi", "ffi")  # ffi
+t = t.replace("ffl", "ffl")  # ffl
+report._track("S3_ligature_expansion", before, t, "ligatures_expanded")
+```
+
+The cycle-12 `decompose_ligatures(t)` call was inserted EARLY in `normalize_text` (~line 1567, just after the NFC step) — it consumes every ligature **before** S3 runs. So S3 now finds nothing, tracks `ligatures_expanded = 0`, and `tests/test_normalization.py::TestFullPipeline::test_report_tracks_changes` fails:
+
+```
+raw = "significant effect −0.73"
+assert report.changes_made.get("ligatures_expanded", 0) > 0   # -> 0, FAIL
+```
+
+That is the 16th pytest failure. **Cycle 12 starved a pre-existing step.**
+
+### The real question to answer first
+
+If `S3_ligature_expansion` already expands ligatures in the normalize body channel, **why did 35 rendered papers still show raw `fi`/`fl` glyphs?** Cycle 12 was triggered by that observation. Possible causes — INVESTIGATE before reworking:
+
+- The rendered `.md` body may come from a path/level that does not run S3 (check what normalization level `render_pdf_to_markdown` applies to body text, and whether `preserve_math_glyphs` or a `NormalizationLevel` branch skips S3).
+- S3 only covers FB00-FB04 — it does **not** handle `ſt` (FB05) / `st` (FB06). Those would survive S3.
+- The ligatures in the render may come from channels that genuinely bypass `normalize_text` entirely: Camelot table cells (`cell_cleaning`), figure/table captions, `unstructured-table` fenced blocks, `raw_text` fallbacks.
+
+### Correct rework (recommended)
+
+1. **Remove** the early `decompose_ligatures(t)` call from `normalize_text` (~line 1567). The body channel already has S3 — do not starve it.
+2. If S3's FB00-FB04-only coverage is the gap, **extend the existing S3 step** to also map `ſt`/`st`→`st` (and keep its `report._track` call intact).
+3. The `cell_cleaning` + `render` post-process `decompose_ligatures` calls are probably the genuine fix (those channels bypass S3) — **verify** by rendering `korbmacher` / `jdm_.2023.16` with ONLY those two calls (no `normalize_text` call) and confirming 0 residual ligatures. Keep them if they close a real gap; the shared helper is fine to keep for those two channels.
+4. Re-confirm `test_report_tracks_changes` passes, run the 26-paper baseline, AI-verify (see JOB 2 — golds now come from article-finder).
+5. **Fix the stale narrative:** the uncommitted `CHANGELOG.md`, `LEARNINGS.md`, `_project/lessons.md`, and `TRIAGE` entries for cycle 12 currently claim a clean *new* fix. They are wrong — cycle 12 duplicated S3. Correct them to describe the actual rework before committing.
+
+**Alternative — clean revert:** `git checkout -- docpluck/ pyproject.toml CHANGELOG.md docs/TRIAGE_2026-05-14_phase_5d_gold_audit.md .claude/skills/docpluck-iterate/LEARNINGS.md .claude/skills/_project/lessons.md` and `rm tests/test_ligature_decomposition_real_pdf.py`, then redo ligature coverage as a fresh, correctly-scoped cycle once the S3-reach question above is answered. This is cleaner than fix-forward if the investigation shows the body path was fine all along.
+
+**Do NOT** ship v2.4.44 until `test_report_tracks_changes` passes and the diff is a genuine, non-duplicate fix.
+
+---
+
+## JOB 2 — Finish the article-finder AI-gold integration
+
+ArticleFinder's full instruction is at:
+`C:\Users\filin\Dropbox\Vibe\ArticleRepository\docs\handoffs\2026-05-16_docpluck_ai-gold-instructions.md`
+**Read it in full.** Summary of what docpluck still owes:
+
+**Already done in session 3 (commit `ac34c7e`):** removed docpluck's private gold-extraction prompt from `references/ai-full-doc-verify.md` (Step 1b); rewrote Step 1 to delegate generation to `article-finder generate-gold`; updated `SKILL.md`, `CLAUDE.md`, `docpluck-qa/SKILL.md`; saved memory `feedback_gold_generation_via_article_finder`.
+
+**Still open — audit `ac34c7e` against ArticleFinder's handoff and adjust:**
+
+1. **Canonical DOI keys are now HARD-ENFORCED.** `ai-gold.py register-view` / `migrate` now *reject* a bare local stem (`chen_2021_jesp`) with an error. Confirm the docpluck-iterate skill (`ai-full-doc-verify.md` "Choosing $KEY" + Phase 5d) resolves the paper's DOI via `ai-gold.py resolve` and passes the DOI — not a stem. If the autonomous loop ever keys by a stem it will now fail loudly.
+2. **`codex` CLI cross-model verification.** `gold-generation.md` now runs an independent Codex / GPT-5.5 model to audit every gold before storage; `generate-gold` blocks without it. `codex-cli 0.128.0` IS installed in this environment — **verify it is authenticated** (`codex --version` works; run `codex login` if calls fail). Note this dependency in the docpluck-iterate skill so a future run does not stall mystifyingly.
+3. **Regenerate the stale `reading` golds.** docpluck's existing cached `reading` golds were produced by the old private prompt and diverge from `gold-generation.md`. Regenerate via `article-finder generate-gold <pdf>`. **Priority — the 3 fragmented papers first:**
+   | Paper | old stem key | canonical DOI key |
+   |---|---|---|
+   | Chen 2021 JESP | `chen_2021_jesp` | `10.1016__j.jesp.2021.104154` |
+   | Xiao 2021 CRSP | `xiao_2021_crsp` | `10.1080__23743603.2021.1878340` |
+   | Efendic 2022 SPPS | `efendic_2022_affect` | `10.1177__19485506211056761` |
+   Then regenerate the rest of docpluck's `reading` golds. After each, confirm with `ai-gold.py views <doi>`.
+4. Run `python ~/.claude/skills/article-finder/ai-gold.py audit` — expect **0 issues**. Coordinate the removal of the old short-stem cache directories with article-finder (do not delete cache data unilaterally — article-finder owns the cache repo's commits).
+
+**Important caveat for JOB 3:** cycles 8-12's Phase-5d verification consumed the *stale* `reading` golds (`tmp/*_gold.md`). The shipped fixes (v2.4.40-43) are still sound — they are keyed on structural signatures and gated by the 26-paper baseline — but once the golds are regenerated, **re-run the Phase-5d verifier for at least efendic / chen / xiao / jdm_m.2022.2 against the fresh golds** to confirm nothing was missed.
+
+---
+
+## JOB 3 — Continue the APA iteration loop
+
+Work queue: `docs/TRIAGE_2026-05-14_phase_5d_gold_audit.md`, section **"SESSION-3 STANDING VERDICT"**. The APA corpus is **NOT clean — ~12 papers still FAIL** Phase-5d on pre-existing defects. Per rule 0e-bis the run continues. Ranked next pickups:
+
+1. **G5b — long-descriptive-title prose guard** (S1, C1, cheap). `render.py::_promote_numbered_subsection_headings` and `_promote_numbered_section_headings` reject headings whose title has a run of ≥5 lowercase-initial words (`max_lc_run >= 5`). This over-rejects legitimate long numbered headings (`4. Knowledge acquisition, decision delay, and choice outcomes`; `2.4.2.2. Inference of planning strategies and strategy types`). For a *numbered* line that already passed the strict regex + (for section-level) the numbering-range/uniqueness/list-adjacency gates, the lc-run guard is near-redundant. Raise the threshold 5→8 in both promoters. ~25 headings, mostly `jdm_.2023.16`. This was the planned cycle 13.
+2. **G5c — split-line numbered headings** (S1, C2). `5.3.`\n\n`Results` — the number alone on a line, the title on the next; renders as an orphan bare-number line, and the content gets a MISLABELED generic `## Results` instead of `### 5.3. Results`. The cycle-3 orphan-arabic-numeral folder's multi-level analogue.
+3. **FIG caption double-emission + truncation** (S2, C2) — ~8 papers.
+4. **G5d — named (unnumbered) heading demotion** (S1, C2-C3) — ~7 papers; section-partitioner work, largest false-positive surface.
+5. **TABLE structure destruction** (S0/S1, C3) — ~11 papers, the single largest blocker; needs a render/structured-extraction coordination *design* — a dedicated session.
+6. **COL column-interleave** (S0, C3) and **GLYPH 011 deleted-minus / efendic `Mchange` no-CI** (S0, C3-C4) — escalate; need the layout channel.
+
+Also queued: a `tests:` regen cycle for the 15 pre-existing pytest failures (all snapshot drift — see `HANDOFF_2026-05-16_iterate_apa_run_3.md` §4). Triage each with a git-stash round-trip before regenerating.
+
+**Iteration discipline (unchanged, non-negotiable):** one defect class per cycle; every fix keyed on a structural signature (never paper identity); 26/26 baseline is the no-regression gate; AI-gold Phase-5d verify every affected paper (gold OBTAINED from article-finder, never self-generated — JOB 2); add a real-PDF regression test in the same cycle; ship incrementally (tagged release per cycle); never report "clean" while corpus FAILs remain (rule 0e-bis).
+
+---
+
+## Run context — what session 3 shipped (cycles 8-11, all clean)
+
+| Cycle | Version | Fix |
+|---|---|---|
+| 8 | v2.4.40 | standalone `2`-for-U+2212 minus recovery via point-estimate∈CI pairing (GLYPH, S0) |
+| 9 | v2.4.41 | numbered subsection-heading regex loosened (trailing dot + internal colon); ~78 `###` headings recovered (G5, S1) |
+| 10 | v2.4.42 | Elsevier page-1 footer (e-mail + ISSN lines) strip (D4, S2) |
+| 11 | v2.4.43 | single-level numbered section-heading promotion (G5a, S1) |
+| — | `ac34c7e` | process fix: gold generation delegated to article-finder (JOB 2 partial) |
+| 12 | (v2.4.44 attempt) | ligature decomposition — **BROKEN, see JOB 1** |
+
+Each of cycles 8-11: 26/26 baseline, 0 new pytest failures, AI-gold verifier OVERALL PASS, real-PDF test added, prod-deployed. Full detail: `docs/HANDOFF_2026-05-16_iterate_apa_run_3.md` and `.claude/skills/docpluck-iterate/LEARNINGS.md`.
+
+`run-meta` (`~/.claude/skills/_shared/run-meta/docpluck-iterate.json`) was left mid-run (verdict blank, `postflight_heartbeat:false`). The fresh session's preflight will re-init it; the session-3 postflight was **not** run (this handoff replaces it). If you want the session-3 signal preserved, note that `bugs_fixed`/`tests_added`/`lessons_appended` arrays in that file already hold cycles 8-12's entries.
+
+## Command cheat-sheet
+
+```
+# 26-paper baseline (the no-regression gate)
+PYTHONUNBUFFERED=1 python -u scripts/verify_corpus.py 2>&1 | awk '{print; fflush()}'
+
+# broad pytest (camelot off)
+DOCPLUCK_DISABLE_CAMELOT=1 python -u -m pytest tests/ -q --tb=line
+
+# render one paper
+python -c "from docpluck.render import render_pdf_to_markdown; from pathlib import Path; print(render_pdf_to_markdown(Path('<pdf>').read_bytes()))"
+
+# AI gold — OBTAIN from article-finder, never self-generate
+python ~/.claude/skills/article-finder/ai-gold.py resolve "<doi>"
+python ~/.claude/skills/article-finder/ai-gold.py check <doi-key> --view reading
+python ~/.claude/skills/article-finder/ai-gold.py get   <doi-key> --view reading
+#   on a miss: invoke the article-finder skill -> generate-gold <absolute-pdf-path>
+python ~/.claude/skills/article-finder/ai-gold.py audit      # expect 0 issues
+
+# prod health
+curl -s https://extraction-service-production-d0e5.up.railway.app/_diag | python -m json.tool
+```
+
+APA test PDFs: `../PDFextractor/test-pdfs/apa/`. Library version files to bump together: `docpluck/__init__.py`, `pyproject.toml`, `docpluck/normalize.py::NORMALIZATION_VERSION` (only if normalize.py changed).

{docpluck-2.4.45 → docpluck-2.4.46}/docs/TRIAGE_2026-05-14_phase_5d_gold_audit.md

@@ -275,6 +275,17 @@ New `render.py::_promote_numbered_section_headings` promotes `N. Title` → `##
 
 `render.py`'s numbered-heading promoters carried a `max_lc_run >= 5` "long lowercase-word run" prose guard that mis-rejected legitimate descriptive headings. Reproduced at HEAD: jdm_.2023.16 alone had **19** multi-level numbered subsection headings demoted to body text, with `max_lc` up to **12** (`3.3.2.1. The quality of planning on the previous trial moderates the effect of reflection`) — far deeper than the TRIAGE's "raise 5→8" estimate. Re-scoped: the lc-run guard is **removed entirely from `_promote_numbered_subsection_headings`** (multi-level dotted numbering + capital-start + no-terminal-punctuation + single ≤80-char line is itself a sufficient heading signature; the lc-run guard cannot distinguish a descriptive heading from prose). For `_promote_numbered_section_headings` (single-level `N.`, real list-collision risk) the guard is kept but raised `5→8`, alongside its existing numbering-range/uniqueness/list-adjacency gates. jdm16: 19 headings recovered; v2.4.44→v2.4.45 diff is heading-promotion only (0 text loss/hallucination); 26/26 baseline.
 
+### Cycle 14 (investigation, run 4) — G5c re-scoped C2 → C3 (no release)
+
+Reproduced G5c at v2.4.45 on jdm_m.2022.2. The TRIAGE framed G5c as "the cycle-3 orphan-arabic-numeral folder's multi-level analogue" (C2, a render-layer fold). Reproduction shows it is **deeper** — section-partitioner work, C3:
+
+- jdm_m.2022.2 has **6** orphan bare-number lines: `5.3.` (L114), `5.4.` (L185), `6.3.` (L260), `6.4.` (L285), `7.3.` (L329), `7.4.` (L403).
+- Only **1** is the clean foldable case the TRIAGE described — `5.4.` immediately followed by a generic `## Discussion` (→ should be `### 5.4. Discussion`). A render-layer fold-into-next-heading handles this one.
+- The other **5** are NOT foldable: pdftotext splits `6.3. Results` into `6.3.\n\nResults`; the section partitioner then consumes the bare word `Results` as a canonical section keyword (building/relocating a `## Results` block elsewhere) and strands `6.3.` with no title and the section body (`We performed one-way ANOVAs…`) starting directly after it. The title word is *gone from that position* — there is no following heading to fold into. `5.3.` (L114) is stranded above `### Figure 1` — same shape.
+- Correct fix layer: `docpluck/sections/` — the partitioner must recognise `N.N.\n\n<CanonicalKeyword>` as ONE numbered heading `N.N. <Keyword>` and not detach the number. That is partitioner work with broad-corpus false-positive surface — a dedicated session, C3.
+
+**Re-scoped:** G5c-1 (render-layer fold of orphan `N.N.` into an adjacent generic `##/###` heading — the `5.4.`/`## Discussion` case, C1-C2) can ship independently; G5c-2 (partitioner-level split-heading rejoin — the 5 title-loss cases, C3) needs the dedicated section-partitioner session alongside G5d and the TABLE cluster.
+
 ### SESSION-3 STANDING VERDICT (rule 0e-bis)
 
 The APA corpus is **NOT clean**. Cycles 8-11 shipped 4 verified incremental fixes (v2.4.40-43), each AI-gold-verified OVERALL PASS with 0 regressions. But ~12 APA papers still FAIL Phase-5d on PRE-EXISTING defects the cycles did not reach. Verifier-confirmed open punch-list:
@@ -282,7 +293,7 @@ The APA corpus is **NOT clean**. Cycles 8-11 shipped 4 verified incremental fixe
 | Defect class | Sev | Papers | Notes |
 |---|---|---|---|
 | **TABLE structure destruction** | S0/S1 | efendic, ar_apa_011, xiao, jdm15/16, chen, maier, ip_feldman (~11) | grid lost → caption-bleed; flat number-dump; empty `<table>` shells; two tables merged; rows dropped. C3 — needs a render/structured coordination design. The single largest blocker. |
-| **G5c split-line numbered headings** | S1 | jdm_m.2022.2 (`5.3.`/`6.3.`/`7.3.` etc.) | number alone on a line, title on the next; renders as orphan bare-number + a MISLABELED generic `## Results`. cycle-3 orphan-folder multi-level analogue. |
+| **G5c split-line numbered headings** — re-scoped run 4 → G5c-1 (render fold, C1-C2) + G5c-2 (partitioner split-heading rejoin, **C3**) | S1 | jdm_m.2022.2 (`5.3.`/`6.3.`/`7.3.` etc.) | number alone on a line; only 1 of 6 cases foldable — the other 5 are partitioner title-loss. See "Cycle 14 (investigation)" above. |
 | **G5d named (unnumbered) heading demotion** | S1 | ar_apa_011 (`Participants`, `Overview`), efendic, chandrashekar, ip_feldman (~7) | section-partitioner work; largest false-positive surface. |
 | ~~**G5b long-descriptive-title prose guard**~~ ✓ FIXED v2.4.45 (cycle 13) | S1 | jdm16, jdm_m2, chen | ~~`≥5-lowercase-word` guard over-rejects legit long numbered headings.~~ Subsection promoter's lc-run guard removed; single-level raised 5→8. |
 | **FIG caption double-emission + truncation** | S2 | jdm_m2, efendic, chan_feldman, ziano, jdm15/16 (~8) | caption inline + in `## Figures` block; truncated mid-word; figure data-labels as orphan body lines. |

{docpluck-2.4.45 → docpluck-2.4.46}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "docpluck"
-version = "2.4.45"
+version = "2.4.46"
 description = "PDF, DOCX, and HTML text extraction and normalization for academic papers"
 readme = "docs/README.md"
 requires-python = ">=3.10"

docpluck-2.4.46/tests/test_orphan_multilevel_number_real_pdf.py

@@ -0,0 +1,134 @@
+"""Regression test for G5c-1 — orphan multi-level section numbers.
+
+The APA Phase-5d sweep found that pdftotext sometimes splits a numbered
+subsection heading such as ``5.4. Discussion`` into a bare ``5.4.`` line and
+a separate ``Discussion`` line. The section partitioner then promotes the
+lone title word to a generic ``## Discussion`` and strands the number::
+
+    5.4.
+    <blank>
+    ## Discussion
+
+Fix (G5c-1): ``_fold_orphan_multilevel_numerals_into_headings`` (render.py
+post-process) — the multi-level analogue of
+``_fold_orphan_arabic_numerals_into_headings`` — folds an orphan ``N.N.``
+number into the immediately-following generic ``##``/``###`` heading and
+emits it at subsection level: ``## Discussion`` → ``### 5.4. Discussion``.
+
+Scope: only the immediately-adjacent case is folded. An orphan number
+separated from its heading by a figure block, or one whose title word the
+partitioner consumed elsewhere (leaving body prose below the number), is
+partitioner-level work (G5c-2) and is intentionally left untouched.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+from pathlib import Path
+
+import pytest
+
+os.environ.setdefault("DOCPLUCK_DISABLE_CAMELOT", "1")
+
+from docpluck.render import (
+    _fold_orphan_multilevel_numerals_into_headings,
+    render_pdf_to_markdown,
+)
+
+TEST_PDFS = Path(__file__).resolve().parents[1].parent / "PDFextractor" / "test-pdfs"
+
+# An orphan multi-level number line immediately followed by a generic
+# (non-numbered, non-Figure/Table) `##`/`###` heading.
+_ORPHAN_MULTILEVEL_BEFORE_HEADING_RE = re.compile(
+    r"(?m)^\d+(?:\.\d+){1,3}\.?[ \t]*\n(?:[ \t]*\n)+#{2,3} (?!\d)(?!Figure\b)(?!Table\b)"
+)
+
+
+# ── Unit tests on the helper ────────────────────────────────────────────
+
+def test_folds_orphan_multilevel_with_dot():
+    assert (
+        _fold_orphan_multilevel_numerals_into_headings("5.4.\n\n## Discussion")
+        == "### 5.4. Discussion"
+    )
+
+
+def test_folds_orphan_multilevel_without_dot():
+    assert (
+        _fold_orphan_multilevel_numerals_into_headings("5.4\n\n## Discussion")
+        == "### 5.4. Discussion"
+    )
+
+
+def test_folds_deeper_numbering_into_h3_heading():
+    assert (
+        _fold_orphan_multilevel_numerals_into_headings("6.1.2.\n\n### Methods")
+        == "### 6.1.2. Methods"
+    )
+
+
+def test_demotes_h2_target_to_subsection_level():
+    # A multi-level number denotes a subsection regardless of the level the
+    # partitioner gave the stranded title — the `##` target becomes `###`.
+    out = _fold_orphan_multilevel_numerals_into_headings("5.4.\n\n## Discussion")
+    assert out.startswith("### ")
+
+
+def test_idempotent_on_already_folded_heading():
+    src = "### 5.4. Discussion"
+    assert _fold_orphan_multilevel_numerals_into_headings(src) == src
+
+
+def test_does_not_fold_before_body_prose():
+    # The non-foldable G5c-2 case: title word consumed elsewhere, body prose
+    # follows the orphan number. Must be left untouched.
+    src = "6.3.\n\nWe performed one-way ANOVAs to test H1a."
+    assert _fold_orphan_multilevel_numerals_into_headings(src) == src
+
+
+def test_does_not_fold_into_figure_heading():
+    # `### Figure N` is a library-emitted structural marker, never a section.
+    src = "5.3.\n\n### Figure 1"
+    assert _fold_orphan_multilevel_numerals_into_headings(src) == src
+
+
+def test_does_not_fold_into_table_heading():
+    src = "5.3.\n\n### Table 1"
+    assert _fold_orphan_multilevel_numerals_into_headings(src) == src
+
+
+def test_does_not_fold_into_already_numbered_heading():
+    src = "5.4.\n\n## 6. Study 2"
+    assert _fold_orphan_multilevel_numerals_into_headings(src) == src
+
+
+def test_leaves_single_level_number_to_the_arabic_folder():
+    # A bare single-level `1.` carries no dot group — the multi-level folder
+    # must not touch it (that is `_fold_orphan_arabic_numerals_into_headings`).
+    src = "1.\n\n## Introduction"
+    assert _fold_orphan_multilevel_numerals_into_headings(src) == src
+
+
+def test_folds_multiple_independent_occurrences():
+    src = "5.4.\n\n## Discussion\n\nbody\n\n6.1.\n\n### Methods"
+    out = _fold_orphan_multilevel_numerals_into_headings(src)
+    assert "### 5.4. Discussion" in out
+    assert "### 6.1. Methods" in out
+
+
+# ── Real-PDF regression test ────────────────────────────────────────────
+
+def test_orphan_multilevel_number_folded_in_render():
+    pdf = TEST_PDFS / "apa" / "jdm_m.2022.2.pdf"
+    if not pdf.exists():
+        pytest.skip(f"fixture missing: {pdf}")
+    md = render_pdf_to_markdown(pdf.read_bytes())
+    # The foldable case: `5.4.` immediately above the generic `## Discussion`.
+    assert "### 5.4. Discussion" in md, "5.4. Discussion subsection not folded"
+    # No orphan multi-level number may sit immediately above a generic heading.
+    leak = _ORPHAN_MULTILEVEL_BEFORE_HEADING_RE.search(md)
+    assert leak is None, (
+        "orphan multi-level number still stranded before a generic heading: "
+        f"{md[leak.start():leak.start() + 48]!r}"
+    )