lumina-wiki 1.7.1 → 1.7.2

package/CHANGELOG.md

@@ -5,6 +5,35 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
 
+## [1.7.2] - 2026-07-05
+
+### Fixed
+
+- `/lumi-ingest` step-03-verify referenced `src/skills/core/verify/` — the
+  repo source tree — instead of the installed workspace path. On any
+  non-Claude-Code IDE target (Codex, Gemini, Cursor, generic), the file
+  never existed post-install, so grounding verification silently had
+  nothing to follow. Fixed all three references to
+  `.agents/skills/lumi-verify/`, which the installer copies unconditionally
+  for every IDE target.
+- Added an `ingest_status` handler for the `not_applicable` verify verdict,
+  which was previously unhandled during ingest.
+
+### Added
+
+- `/lumi-ingest` now checks external identifiers (DOI/arxiv/S2) for an
+  existing source page before generating a slug, so the same paper
+  ingested under a different title no longer creates a duplicate page.
+- Concept stub creation now scans existing concepts for acronym/expansion
+  and singular/plural variants before creating a new one.
+- Key Claims in drafted source pages now require a source locator
+  (section/page/heading), so the grounding reviewer in step-03 no longer
+  has to re-scan the whole raw file to check a claim.
+- A concept-count rubric (roughly 3-7 per source) to keep the graph from
+  being diluted by over-extracted keyword stubs.
+- PDF preprocessing now runs before type detection in step-01, so runtimes
+  without native PDF reading don't fail attempting to read the raw binary.
+
 ## [1.7.1] - 2026-07-02
 
 ### Added

package/package.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/package.json",
   "name": "lumina-wiki",
-  "version": "1.7.1",
+  "version": "1.7.2",
   "description": "Domain-agnostic, multi-IDE wiki scaffolder — Karpathy's LLM-Wiki vision, cross-platform and pack-based.",
   "keywords": [
     "llm-wiki",

package/src/skills/core/ingest/references/dedup-policy.md

@@ -3,6 +3,31 @@
 Use this reference before creating or updating source, concept, person, and graph
 records during `/lumi-ingest`.
 
+## Identifier Dedup (Before Slug Generation)
+
+Slug comparison alone misses duplicates: the same paper ingested under two title
+variants (with/without subtitle, arxiv version suffix, translated title) produces
+two different slugs. External identifiers are the stronger key, so check them
+first whenever one is known (from `resolve_pdf.py` output, or parsed from the
+input URL).
+
+```bash
+# One grep per known identifier, over source-page frontmatter.
+# Use the normalized value: bare DOI (10.x/y), bare arxiv id (2604.03501 — no version suffix).
+grep -rln "10.48550/arxiv.2604.03501" wiki/sources/
+grep -rln "2604.03501" wiki/sources/
+```
+
+- **Hit** — an existing page carries the same identifier. This run is a re-ingest
+  of that page's slug, regardless of what slug the current title would generate.
+  Confirm with the user, then follow the re-ingest path below; do not create a
+  second page.
+- **No hit** — proceed to slug generation.
+
+A plain substring grep is deliberate: identifiers are unique enough that false
+positives are rare, and a rare false positive only costs one confirmation
+question. A false negative (skipping the check) costs a duplicate source page.
+
 ## Source Pages
 
 Generate the source slug with:
@@ -47,7 +72,22 @@ making any `add-edge concepts/<slug>` calls.
 
 ## Concept And Person Stubs
 
-Before creating a concept or person page, check metadata:
+Exact-slug lookup only catches exact matches — `rlhf` and
+`reinforcement-learning-from-human-feedback` would become two pages. Once per
+ingest run, list the existing concepts and hold the list in mind while creating
+stubs:
+
+```bash
+node _lumina/scripts/wiki.mjs list-entities --type concept
+```
+
+For each candidate concept, scan that list for variants of the same term:
+acronym vs expansion, singular vs plural, hyphenation or word-order differences.
+If a variant exists, link to the existing slug instead of creating a new stub —
+and if unsure whether two terms are the same concept, ask the user rather than
+guessing.
+
+Then, before creating a concept or person page, check metadata:
 
 ```bash
 node _lumina/scripts/wiki.mjs read-meta concepts/<slug>

package/src/skills/core/ingest/references/step-01-draft.md

@@ -6,7 +6,7 @@
 - Never modify files in `raw/`. Read-only.
 - Never hand-edit `wiki/graph/edges.jsonl` or `wiki/graph/citations.jsonl`; use `wiki.mjs add-edge` and `wiki.mjs add-citation`.
 - Never overwrite an existing wiki page without user confirmation.
-- All frontmatter writes go through `wiki.mjs set-meta`. Never write to `wiki/*.md` directly. (Body text goes through Write tool atomically; finalizing frontmatter still uses `set-meta`.)
+- Frontmatter discipline, two cases: (a) **initial creation** of a page that does not exist yet — Write the whole file (frontmatter + body) in one shot from the template; (b) **any change to an existing page's frontmatter** — always `wiki.mjs set-meta`, never Write/Edit on the frontmatter block. Body text of an existing page may be edited with Write/Edit; the frontmatter block may not.
 - `raw/tmp/` accepts additions only; never overwrite a file there.
 - `raw_paths` must list permanent artifacts only. Reject `raw/tmp/*` entries.
 - Keep a phase-level checkpoint after every phase — an interrupted run must resume cleanly.
@@ -69,9 +69,13 @@ node _lumina/scripts/wiki.mjs checkpoint-read research-discover shortlist
 
 Match title to a shortlist entry, extract URL, fall through to Mode B.
 
+Fallback: if the checkpoint does not exist (research pack not installed, or no discover run yet) or no shortlist entry matches the title, do not guess. Ask the user for a URL or identifier (arxiv ID / DOI), then fall through to Mode B with their answer.
+
 ### Phase 1 — Detect type
 
-Read first ~200 lines of the source. Classify:
+If the source is a PDF or too large to read comfortably in one pass, follow `pdf-preprocessing.md` **before reading anything** — on runtimes without native PDF reading, a direct Read of the binary fails here, not in Phase 3.
+
+Read first ~200 lines of the source (or of the extracted text). Classify:
 - "Abstract", "Introduction", "References" → `paper`
 - Chapter structure → `book`
 - Web byline + publication → `article`
@@ -82,6 +86,8 @@ Write checkpoint: `phase: "detect"`.
 
 ### Phase 2 — Generate slug
 
+**Identifier dedup comes first.** If any external identifier is known at this point (DOI / arxiv ID / S2 ID — from Mode B resolution or parsed from the input URL), check for an existing source page carrying the same identifier BEFORE generating a slug — title variants produce different slugs, so slug comparison alone misses duplicates. See `dedup-policy.md` § Identifier Dedup. On a hit, this run is a re-ingest of the matched slug: confirm with the user and skip new-page creation.
+
 ```bash
 node _lumina/scripts/wiki.mjs slug "<Title>"
 ```
@@ -107,7 +113,7 @@ Write checkpoint: `phase: "slug"` (already included in the merge above).
 
 ### Phase 3 — Write source page
 
-For PDFs / large sources, follow `pdf-preprocessing.md` first.
+For PDFs / large sources, the extraction from Phase 1 (`pdf-preprocessing.md`) applies here too — draft from the extracted text, section by section for long sources.
 
 Draft `wiki/sources/<slug>.md` from `_lumina/schema/page-templates.md` Source template. Required frontmatter: `id`, `title`, `type`, `created`, `updated`, `authors`, `year`, `importance`, `provenance`. Optional but encouraged: `urls`, `raw_paths`, `confidence`, `external_ids`.
 
@@ -131,6 +137,10 @@ node _lumina/scripts/wiki.mjs set-meta sources/<slug> sources "[$entry]" --json-
 
 Required body sections: `## Summary` (2–4 sentences), `## Key Claims` (bulleted, with confidence), `## Concepts` (`[[concept-slug]]` links), `## People` (`[[person-slug]]` links), `## Open Questions`.
 
+`## Key Claims` — each claim ends with a locator pointing at where the source supports it: a section anchor, page, or heading, e.g. `(§3.2)`, `(p. 5)`, `(Table 1)`. The grounding check in step-03 reads these locators to find evidence; a claim without one forces the reviewer to re-read the whole raw file and weakens the check. If a claim synthesizes multiple places, list the primary locator.
+
+`## Concepts` — be selective, not exhaustive. A term earns a concept link only if (a) it is central to this source's contribution, or (b) it is likely to recur across other sources in this wiki. Aim for roughly 3–7 concepts per source; minor keywords belong in body prose, not as concept links. Every linked concept implies a stub + edges in Phases 4–5, so over-extraction dilutes the graph.
+
 Provenance rubric (raw-centric):
 - `replayable` — `raw_paths` non-empty, every entry resolves to existing file
 - `partial` — `urls` has entries but `raw_paths` empty or missing
@@ -156,7 +166,7 @@ node _lumina/scripts/wiki.mjs resolve-alias "<concept-name>"
 
 If it resolves to a foundation, link via `[[foundations/<slug>]]` and add `grounded_in` edge instead of creating a stub. See `dedup-policy.md` § Foundation Resolution.
 
-Apply `dedup-policy.md` before creating/updating stubs. Existing pages are updated conservatively.
+Apply `dedup-policy.md` before creating/updating stubs — including the concept variant scan (acronym vs expansion, singular vs plural), which catches duplicates that exact-slug lookup misses. Existing pages are updated conservatively.
 
 New stubs use the templates in `_lumina/schema/page-templates.md`.
 
@@ -187,10 +197,14 @@ node _lumina/scripts/wiki.mjs add-citation sources/<slug> sources/<cited-slug>
 
 Do not create stubs for cited sources not yet ingested — note them in `## Open Questions`.
 
+Write checkpoint: `phase: "citations"`.
+
 ### Phase 7 — Update wiki/index.md
 
 Add the new source page (and any new concept/person pages) to the catalog between `<!-- lumina:index -->` markers. Format: `- [[sources/<slug>]] — <one-line description>`.
 
+Write checkpoint: `phase: "index"`.
+
 ## Draft Gate
 
 Present a draft summary to the user:
@@ -211,7 +225,7 @@ Use the user's configured communication language. Explain "provenance", "edges",
   ```
   → NEXT
 - **E**: Take the user's revision instructions. Re-edit the affected files (source page, stubs, or edges as instructed). Re-present the draft summary. Loop back to "HALT and ask human" — do not advance.
-- **Q**: Leave the phase-level checkpoint in place; do not write `ingest_status`. **STOP — do not read the NEXT directive below.** Exit cleanly with no further action this run. The next `/lumi-ingest <slug>` invocation resumes from this point.
+- **Q**: Leave the phase-level checkpoint in place; do not write `ingest_status`. Before exiting, tell the user in plain language that the draft pages and links written so far stay in the wiki as work-in-progress — coming back to `/lumi-ingest <slug>` continues from here, and `/lumi-reset` can clean up if they decide not to keep this source at all. **STOP — do not read the NEXT directive below.** Exit cleanly with no further action this run.
 
 ## NEXT
 

package/src/skills/core/ingest/references/step-03-verify.md

@@ -3,7 +3,7 @@
 ## RULES
 
 - Read `README.md` at the project root before this step if you have not already in this session.
-- Reuse the existing `/lumi-verify` skill in grounding-only mode. Do not inline a duplicate grounding reviewer here — single source of truth for grounding logic lives in `src/skills/core/verify/`.
+- Reuse the existing `/lumi-verify` skill in grounding-only mode. Do not inline a duplicate grounding reviewer here — single source of truth for grounding logic lives in `.agents/skills/lumi-verify/`.
 - All frontmatter writes go through `wiki.mjs set-meta`. Never write to `wiki/*.md` directly.
 - Drift is a hard halt: a missing `raw_paths` file is a stronger signal than any finding and should not be silently downgraded.
 - This step asks the user only when there are source-check findings, missing source files, or a confidence downgrade. If the source check passes, continue automatically.
@@ -24,7 +24,7 @@ Invoke `/lumi-verify` on the entry restricted to the grounding reviewer. Three r
 
 **Tier 1 — Agent tool available (Claude Code):**
 
-Spawn a sub-agent with the verify SKILL prompt and the slug, instructing it to run grounding only (skip blind, skip external). Wait for completion, then read the writeback:
+Spawn a sub-agent instructed to read and follow `.agents/skills/lumi-verify/SKILL.md` with this slug as the target, grounding only (skip blind, skip external). The sub-agent is deliberate: it activates the verify skill in a blank context, so the verdict is not anchored by the reasoning chain that just drafted the pages. Do not invoke the verify skill inline in this conversation (e.g. via a skill-invocation tool) — same-context verification inherits drafting bias. Wait for completion, then read the writeback:
 
 ```bash
 node _lumina/scripts/wiki.mjs read-meta sources/<slug>
@@ -32,9 +32,9 @@ node _lumina/scripts/wiki.mjs read-meta sources/<slug>
 
 **Tier 2 — Bash-only runtime (Codex, Gemini, Cursor, generic):**
 
-Read fully and follow `src/skills/core/verify/SKILL.md` Grounding pipeline (Section: "Grounding reviewer"), with this slug as the target. The verify skill's writeback contract is the same — `verify_status` and `findings` written to the entry frontmatter. After the grounding pass returns, control returns to this step's Phase 8.6.
+Read fully and follow `.agents/skills/lumi-verify/SKILL.md` Grounding pipeline (Section: "Grounding reviewer"), with this slug as the target. The verify skill's writeback contract is the same — `verify_status` and `findings` written to the entry frontmatter. After the grounding pass returns, control returns to this step's Phase 8.6.
 
-If `src/skills/core/verify/references/reviewers.md` exists, it is the canonical Grounding reviewer prompt; load it as part of following the verify SKILL.
+If `.agents/skills/lumi-verify/references/reviewers.md` exists, it is the canonical Grounding reviewer prompt; load it as part of following the verify SKILL.
 
 **Tier 3 — User opts out:**
 
@@ -54,6 +54,7 @@ Inspect `verify_status` and `findings`:
 | `findings_pending` | Patch/defer findings written | Present findings inline; user chooses A/E/W/Q |
 | `drift_detected` | `raw_paths` resolves to missing files | **Hard HALT** — do not present standard menu; force user to repair raw or set `provenance: missing` explicitly |
 | `skipped` | Tier 3 opt-out | Write verified status and continue automatically; the user already opted out |
+| `not_applicable` | Verify refused: target is not a `sources/*` entry | Should never happen during ingest (the target is always `sources/<slug>`). Treat as an internal error: HALT, report the slug that was passed to verify, and do not advance `ingest_status`. |
 
 For `passed`, tell the user in one short sentence that the page matched the source closely enough to continue. Then write `ingest_status: verified` and go to NEXT: