cite-citadel 0.1.0__tar.gz

cite_citadel-0.1.0/.claude/skills/verify-example/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: verify-example
+description: End-to-end test of the whole citadel ingest pipeline on the bundled coffee+tea example corpus — ingest raw/ into a fresh wiki, run the structural gates (citadel check + lint), then grade the result against the ground-truth answer key (planted contradictions, repetitions, single-source facts, the one deliberately-false fact, cross-topic links). Use this whenever the user wants to run the e2e or example test, verify or grade the example corpus, (re)build the demo wiki, prove that citations and contradictions still surface, or check that a change to ingest, the rules tree (citadel/rules/), the ingest prompts, or the store still folds the corpus correctly — even if they do not say the word "skill".
+---
+
+# Verify the example corpus end-to-end
+
+The `raw/` corpus (5 coffee + 5 tea files) is **designed** to stress the three guarantees: facts
+repeat, contradict, hide in one place, vary in style, name fictional people, and include one
+flat-out-false claim. The answer key is `.claude/skills/verify-example/ground-truth.md` — the ingest
+never sees it (it lives outside `raw/`/`wiki/`/`docs/`). This skill runs the real pipeline, then grades
+the wiki against that key. All paths are relative to the repo root.
+
+This is a **heavy, intentional** test: Mode A shells out to the LLM ingest CLI (slow, uses your
+subscription). For fast iteration on the grader, use Mode B.
+
+## Preconditions
+
+- The ingest CLI is installed and logged in (default `claude`; run `claude` once and `/login`). Mode B
+  needs no CLI.
+- The 10 example files are present: `ls raw/*.md | wc -l` → should be **10**.
+- Prefer a clean-ish git tree so a stray edit is easy to see (`git status`).
+
+## Mode A · Full E2E (ingest + grade)
+
+Regenerates the wiki from scratch, so it both **rebuilds the showcase wiki** and tests the pipeline.
+
+**1 · Start from an empty wiki** (so every source is ingested fresh — the manifest makes ingest
+idempotent, so a leftover wiki/manifest would skip everything). Move the current wiki aside rather than
+deleting it:
+
+```bash
+[ -d wiki ] && mv wiki "/tmp/citadel-wiki-bak.$(date +%s)" || true
+```
+
+**2 · Ingest the whole corpus** (one agentic session per file; minutes, not seconds):
+
+```bash
+time uv run python -m citadel ingest
+```
+
+Expected: a report ending `... created, ... updated, 0 errors` and **no** "WARNING — broken
+cross-links". 10 sources processed. If a source errored (missing/again-not-logged-in CLI, timeout), fix
+that first — the grade is meaningless on a partial wiki.
+
+**3 · Structural gates** (hard pass/fail, pure code — see ground-truth §G):
+
+```bash
+uv run python -m citadel check        # expect: "OK — no validation issues."
+uv run python -m citadel lint          # expect: final line "OK"
+```
+
+A non-zero `lint` (missing type / broken link / fabricated source / `[[wikilink]]`) or any `check`
+error is an automatic FAIL — the pipeline produced a structurally invalid wiki.
+
+**4 · Grade against the answer key** — read `ground-truth.md` in full, then judge each section against
+the wiki using the greps below as evidence (do not grade from memory). Then go to **Grading**.
+
+**5 · Keep or restore the wiki.** The freshly built `wiki/` is the new showcase. To keep it, leave it
+(and `git add wiki/` when committing). To revert: `rm -rf wiki && mv /tmp/citadel-wiki-bak.* wiki`.
+
+## Mode B · Grade-only (no re-ingest)
+
+Grades the wiki that is already on disk — for iterating on the grader/ground-truth or re-checking after
+a manual wiki edit. Skip steps 1–2; run **3** (gates) and **4** (grade) only.
+
+## Grading — evidence commands
+
+Run these and judge the output against `ground-truth.md`. Page **names** are LLM-chosen and vary run to
+run — **grade by content, not by filename**.
+
+```bash
+# C · contradictions surfaced (target >=2 of 4; stretch 4/4)
+grep -rn "CONTRADICTION" wiki/ | grep -v index.md
+# the four subjects to confirm by eye (one value vs the other, or a callout):
+grep -rni "green tea" wiki/ | grep -iE "28|50 ?mg"
+grep -rni "half-life" wiki/ | grep -iE "3 ?h|5 ?h|hour"
+grep -rni "aurora" wiki/ | grep -iE "198[57]"
+grep -rni "thornbury\|1650\|1657" wiki/
+
+# D · the false fact present, attributed, and questioned (not silently fixed/dropped)
+grep -rni "caffeine-free\|midnight\|burns off" wiki/      # the claim must appear
+grep -rn  "\[\^llm" wiki/                                  # ideally an LLM caveat near it
+uv run python -m citadel lint | grep -A20 "model-supplied"  # pages carrying [^llm] facts
+
+# E · the subtle "don't drop me" fact survived
+grep -rni "cold brew" wiki/ | grep -iE "higher|more caffeine|ratio|steep"
+
+# A · single-source facts survived
+grep -rni "l-theanine" wiki/ ; grep -rni "ceremonial\|culinary" wiki/ ; grep -rni "9 ?bar\|63 ?mg" wiki/
+
+# B · repetitions merged, not duplicated — there should NOT be one isolated page per raw file
+grep -rln "95 ?mg" wiki/        # the 95 mg fact should live on ~one page, co-cited, not many
+grep -rln "twice the caffeine\|2x\|2× caffeine" wiki/
+
+# F · cross-topic bridge: coffee<->tea connected, Thornbury reachable from both
+grep -rni "caffeine" wiki/ | grep -i "tea" | grep -i "coffee"
+grep -rni "thornbury" wiki/
+
+# H · abbreviations: TDS/EGCG spelled-out-once carried in (defined); EY flagged undefined
+grep -rni "total dissolved solids\|(TDS)\|(EGCG)\|epigallocatechin" wiki/   # expansions preserved
+grep -rln "type: Abbreviation" wiki/ 2>/dev/null || true                     # bonus: a glossary page
+uv run python -m citadel lint | grep -A12 -i "undefined abbrev"              # EY should appear; TDS/EGCG should NOT
+
+# provenance density (every fact cited)
+grep -rno "\[\^s[0-9]" wiki/ | wc -l       # many raw citations
+uv run python -m citadel search "caffeine"  # the search seam works
+```
+
+Optional visual check: `uv run python -m citadel view --no-open` then open the printed `file://` path.
+
+## Pass criteria
+
+Report a table of: **hard gates** (all must hold) and **soft checks** (report caught/partial/missed,
+don't hard-fail a single miss). From `ground-truth.md`:
+
+- **Hard:** `check` 0 errors · `lint` OK · all §A single-source facts present · §D false claim present +
+  attributed (NOT silently corrected) · §E subtle fact present · §F coffee and tea not two disconnected
+  islands · §B not one-isolated-page-per-raw-file.
+- **Soft:** §C contradictions surfaced (≥2 good, 4/4 great) · §D carries an explicit `[^llm]` caveat ·
+  §B merges maximally tidy · §H abbreviations (TDS/EGCG carried in with their expansion and not flagged
+  undefined; EY surfaced by lint's undefined-abbreviations check).
+
+End with a one-line verdict and, if anything failed, the specific guarantee it breaks (organized /
+links / provenance) and the file/fact involved.
+
+## Gotchas
+
+- **Ingest is non-deterministic.** Page filenames, exact wording, and which contradictions get a
+  `> [!CONTRADICTION]` callout vary between runs and models. Grade semantics (is the fact present, cited,
+  merged?), never exact paths. A contradiction missed on one run that was caught before is a *soft*
+  regression worth noting, not a hard fail.
+- **The manifest makes ingest skip unchanged sources.** If you forget step 1 (empty wiki) the run does
+  nothing and the grade reflects the *old* wiki. Always start Mode A from a moved-aside wiki.
+- **A wiki outside the repo root breaks citations.** Do NOT point `CITADEL_WIKI_DIR` at `/tmp` for this —
+  the `[^s..]` links are `../../raw/...` relative and must resolve to the repo's `raw/`. Keep `wiki/` at
+  the repo root (sibling of `raw/`); only the *backup* goes to `/tmp`.
+- **The false fact is supposed to be in the wiki.** Do not "fix" it. A wiki that silently states the
+  truth instead, with no `aurora-coffee-blog.md` citation, is a provenance FAIL, not a pass.
+- **Fictional entities are not errors.** Caffè Aurora, Lina Marchetti, Thornbury & Lin etc. are invented
+  on purpose; the wiki recording them is correct.
+- **Model matters.** A weaker `CITADEL_INGEST_MODEL` catches fewer contradictions and adds fewer `[^llm]`
+  caveats. Note the model (it is recorded per source in `wiki/.citadel_ingested.json` and the report) so
+  soft-score comparisons are apples-to-apples.
+
+## Troubleshooting
+
+- `ingest` reports a per-source error about the CLI → it is missing or not logged in; `claude` then
+  `/login`, or set `CITADEL_LLM_CLI`/`*_CLI_PATH`. Re-run from step 1.
+- `check`/`lint` fail right after a green ingest → the agent introduced a broken cross-link or skipped a
+  required field; read the issue, it names the page. This is a real pipeline finding, report it.
+- Grade looks empty / everything "missing" → you are likely grading a stale or empty `wiki/`; confirm
+  `ls wiki/**/*.md` shows pages and that step 2 actually processed 10 sources.

cite_citadel-0.1.0/.claude/skills/verify-example/ground-truth.md

@@ -0,0 +1,142 @@
+# Ground truth — the coffee+tea example corpus
+
+This is the **answer key** for the `verify-example` end-to-end test. The `raw/` corpus is fed to
+`citadel ingest`; this file is **not** — it lives under `.claude/` (outside `raw/`/`wiki/`/`docs/`), so
+the ingest pipeline never sees it. The skill reads it to grade the wiki the pipeline produced.
+
+The corpus is deliberately messy — facts repeat, contradict, hide in one place, vary in writing style,
+and include invented people and one flat-out-false claim — so a clean pass exercises all three of the
+project's guarantees: **stays organized**, **links keep working**, **honest provenance**.
+
+> Some people/companies below are **fictional by design** (Caffè Aurora, Lina Marchetti, Thornbury & Lin,
+> Sir Edmund Thornbury, Mei Lin). They are *not* errors — the wiki should record them faithfully as the
+> sources state them.
+
+## The 10 raw files
+
+| file | topic | register | gist |
+| ---- | ----- | -------- | ---- |
+| `raw/coffee-guide.md` | coffee | structured reference | species, origin, processing, roast, ratios, caffeine |
+| `raw/espresso-and-cafe-culture.md` | coffee | prose essay | espresso mechanics; Lina Marchetti founds Caffè Aurora (1987) |
+| `raw/cold-brew-notes.md` | coffee | lab notebook | cold-brew method; "cold brew higher caffeine"; half-life ~3 h |
+| `raw/coffee-health-faq.md` | coffee | FAQ | half-life ~5 h, 95 mg, robusta 2×, adenosine, pregnancy |
+| `raw/aurora-coffee-blog.md` | coffee | brand blog | Caffè Aurora (1985); **the false "dark roast = caffeine-free" claim** |
+| `raw/tea-guide.md` | tea | structured reference | Camellia sinensis, oxidation, temps, caffeine (green 28 mg) |
+| `raw/tea-history-and-trade.md` | tea | prose narrative | tea trade; Thornbury & Lin (1657); green tea ~50 mg |
+| `raw/matcha-and-preparation.md` | tea | how-to | matcha prep; 60–70 mg; ceremonial vs culinary grade |
+| `raw/tea-health-faq.md` | tea | FAQ | L-theanine; EGCG; tea-vs-coffee caffeine compare |
+| `raw/thornbury-tea-blog.md` | tea | brand blog | Thornbury & Lin (1650); tea-vs-coffee rivalry |
+
+## A · Known facts that MUST appear in the wiki (cited to the right source)
+
+Single-source facts (only one file states them — they must **survive** ingest, not be dropped):
+
+| fact | source file | note |
+| ---- | ----------- | ---- |
+| Espresso pulled ~9 bar, ~25–30 s, ~18 g in → ~36 g out, crema; ~63 mg/shot | `espresso-and-cafe-culture.md` | espresso mechanics |
+| Cold brew often **higher** caffeine than hot drip (ratio + long steep) | `cold-brew-notes.md` | the "easy to drop" fact — see §D |
+| Matcha grades: ceremonial vs culinary | `matcha-and-preparation.md` | |
+| Matcha ~60–70 mg (whole leaf consumed) | `matcha-and-preparation.md` | |
+| **L-theanine** (calm-alert, ~unique to tea) | `tea-health-faq.md` | |
+| Coffee 3–4 cups/day *associated with* lower type-2-diabetes / Parkinson's risk | `coffee-health-faq.md` | must stay an *association* |
+| White tea contains caffeine (~30–55 mg) | `tea-guide.md` | |
+| Caffè Aurora founder Lina Marchetti, Trieste | `espresso-and-cafe-culture.md` + `aurora-coffee-blog.md` | year contradicts (§C) |
+| Thornbury & Lin (Edmund Thornbury, Mei Lin), London/Canton, traded tea **and** coffee | `tea-history-and-trade.md` + `thornbury-tea-blog.md` | year contradicts (§C) |
+
+## B · Repetitions — must MERGE / co-cite, never duplicate one-page-per-file
+
+| fact | files | expectation |
+| ---- | ----- | ----------- |
+| Robusta ≈ 2× the caffeine of Arabica | `coffee-guide.md` + `coffee-health-faq.md` | one statement, both cited (`[^s..]` ×2) |
+| Drip coffee (8 oz) ≈ 95 mg | `coffee-guide.md` + `coffee-health-faq.md` | one statement, both cited |
+| Green-tea brew temp 70–80 °C | `tea-guide.md` + `matcha-and-preparation.md` | merged |
+| All true tea = *Camellia sinensis* | `tea-guide.md` + `matcha-and-preparation.md` (+ history) | merged |
+
+Failure mode to catch: a `concepts/coffee-guide.md` AND a `concepts/coffee-health-faq.md` that each
+restate the 95 mg / robusta facts in isolation = the pipeline made one-page-per-file instead of routing
+by fit. (Page **names** are LLM-chosen and may differ — judge by content, not filename.)
+
+## C · Contradictions — must surface as `> [!CONTRADICTION]` (or both conflicting values co-cited on one page)
+
+| id | subject | source A | source B |
+| -- | ------- | -------- | -------- |
+| `green-tea-caffeine` | green-tea caffeine per cup | `tea-guide.md`: ~28 mg | `tea-history-and-trade.md`: ~50 mg |
+| `half-life` | caffeine half-life | `coffee-health-faq.md`: ~5 h | `cold-brew-notes.md`: ~3 h |
+| `aurora-year` | Caffè Aurora founding year | `espresso-and-cafe-culture.md`: 1987 | `aurora-coffee-blog.md`: 1985 |
+| `thornbury-year` | first English tea import | `tea-history-and-trade.md`: 1657 | `thornbury-tea-blog.md`: 1650 |
+
+A passing wiki surfaces these rather than silently picking one value. Catching all four is the
+**stretch** goal; catching ≥2 and never *silently* overwriting is the **hard** goal. Report each as
+caught / not-caught.
+
+## D · The one blatantly false fact (single source → must stand, but flagged)
+
+`aurora-coffee-blog.md` asserts: **"Aurora Midnight is a dark roast, and the roasting fire burns off
+the caffeine, so Midnight is caffeine-free — the darker the roast, the less caffeine."** This is false
+in reality, but it is the blog's own word and no other source makes the same Aurora-Midnight claim.
+
+Required wiki behaviour (honest provenance):
+1. The claim **appears**, cited to `aurora-coffee-blog.md` (`[^s..]`) — not silently dropped.
+2. It is **not** presented as unqualified truth: ideally an `[^llm]` model-knowledge note questions it,
+   or a `> [!CONTRADICTION]` ties it to `coffee-guide.md`'s "roast barely changes caffeine / dark roast
+   is not decaffeinated."
+3. It is **not** rewritten into the correct fact without attribution (that would be inventing/erasing).
+
+`citadel lint` lists pages carrying `[^llm]` facts — a good signal this was handled.
+
+## E · The subtle "must-not-be-dropped" fact
+
+`cold-brew-notes.md` states cold brew often ends up **higher** in caffeine than hot drip (high
+grounds-to-water ratio + long steep), against the "cold = less caffeine" assumption. It is mentioned
+once, in a notebook among other detail — easy to lose. It **must be present** in the wiki.
+
+## F · Cross-topic bridges (coffee ↔ tea)
+
+- **Caffeine** is the strongest link: both topics give numbers (drip 95 mg vs black tea 47 mg; tea
+  generally less than coffee). Expect either a shared caffeine `Concept` page cited by both topics, or
+  dense cross-links between the coffee and tea caffeine material. The two topics must **not** end up as
+  two disconnected islands.
+- **Thornbury & Lin** traded tea **and** coffee → its page (likely `type: Organization`) should be
+  reachable from both the tea trade material and a coffee-trade mention.
+
+## G · Structural gates (hard pass/fail — pure code, no judgement)
+
+- `citadel check` → **0 errors** (required fields, honest/defined citations, relative non-broken links).
+- `citadel lint` → **OK** (no missing-type, no broken links, no fabricated sources, no `[[wikilinks]]`).
+- Every factual sentence carries a `[^s..]` (raw) or `[^llm..]` (model) marker; every `[^s..]` resolves
+  to a real `raw/` file.
+- Pages routed by `type` into the right folder (`concepts/`, `organizations/`, `persons/`, …).
+
+## H · Abbreviations (glossary + the undefined-abbreviation lint check)
+
+The corpus seeds the abbreviation machinery on purpose:
+
+- **Spelled out once → defined.** `TDS` (Total Dissolved Solids) and `EGCG` (epigallocatechin gallate)
+  are each expanded **exactly once**, in parenthetical form, then used bare elsewhere (`TDS` in
+  `cold-brew-notes.md` + `coffee-guide.md`; `EGCG` in `tea-guide.md` + `tea-health-faq.md`). The wiki
+  should carry that expansion — inline (`Total Dissolved Solids (TDS)`) and/or as a `type:
+  Abbreviation` glossary page — so `citadel lint` does **not** list TDS/EGCG as undefined, and they
+  may appear in the generated `## Abbreviations` table in `index.md`.
+- **Never spelled out in raw → defined honestly OR flagged.** `EY` (extraction yield) is used across
+  the coffee brewing material (`cold-brew-notes.md` + `coffee-guide.md`) but **never** expanded in any
+  raw file. Two honest outcomes are acceptable: (a) the wiki leaves it bare and `citadel lint`'s
+  *Undefined abbreviations* check surfaces `EY`; or (b) — observed, and preferable — the agent creates a
+  `type: Abbreviation` page that expands it (`EY — Extraction Yield`) and labels the expansion `[^llm]`,
+  since that expansion is model knowledge, not from a raw file. Either way `citadel lint` should surface
+  **at least one** undefined abbreviation (`EY` and/or `FAQ`); the build never fails on it (advisory).
+
+This is what the user means by "abbreviations should appear, at least one spelled out once": TDS/EGCG
+are the spelled-out-once ones (→ defined, ideally a glossary page); EY is never spelled out in raw, so
+the pipeline must either flag it (lint) or define it with an honest `[^llm]` expansion — and never just
+silently invent the expansion as if it came from the source.
+
+## Scoring
+
+**Hard gates** (must all hold): §G structural, §A single-source facts all present, §D claim present +
+attributed (not silently corrected), §E subtle fact present, §F not two disconnected islands.
+
+**Soft / probabilistic** (LLM-dependent — report, don't hard-fail on a single miss): §C contradiction
+callouts (target ≥2 of 4, stretch 4/4), §B merges being maximally tidy, §D carrying an explicit `[^llm]`
+caveat, §H abbreviations (TDS/EGCG carried in with their expansion and NOT flagged undefined; EY
+surfaced by lint's undefined-abbreviations check). Report each soft check as caught/partial/missed so
+regressions are visible across runs.

cite_citadel-0.1.0/.env.example

@@ -0,0 +1,4 @@
+# .env.example — moved
+
+Moved to [citadel/templates/env.example](citadel/templates/env.example) — packaged with the wheel as the
+`citadel init` .env template; edit there.

cite_citadel-0.1.0/.github/copilot-instructions.md

@@ -0,0 +1,162 @@
+# GitHub Copilot instructions — cite-citadel
+
+Repository guidance for GitHub Copilot. This mirrors [`CLAUDE.md`](../CLAUDE.md); keep the two in
+sync when either changes.
+
+## What this is
+
+`cite-citadel` (CLI: `citadel`, PyPI package: `cite-citadel`) is an LLM-maintained, fully-cited
+personal wiki in Google's [Open Knowledge Format](../docs/okf-reference.md), with an MCP server so an
+AI can search and read it. It implements Karpathy's LLM-Wiki pattern: drop arbitrary text-bearing
+files into `raw/`, and one agentic CLI session per source folds each into a cross-linked OKF wiki
+under `wiki/`. Pure Python 3.12, KISS. Runtime deps are only `mcp` and `pyyaml` — **there is no LLM
+SDK and no API key**: ingest shells out to a coding-agent CLI you already have logged in
+(`claude`/`copilot`/`gemini`).
+
+## Commands
+
+Setup: `uv sync` (creates `.venv`, installs deps + the `dev` group + the `citadel` script).
+
+Use the **portable** invocation everywhere — it works identically on Linux/macOS/Windows and needs
+no `.exe` (the `uv run citadel …` shorthand often breaks on Windows because AV quarantines uv's
+generated `citadel.exe`):
+
+```bash
+uv run python -m citadel <subcommand>
+```
+
+Subcommands: `init [DIR]` (scaffold a workspace: `citadel.toml` marker, `.env`, `raw/`, `wiki/`;
+idempotent), `ingest [paths…]` (fold raw/ into the wiki; `--verbose`/`-v` streams the agent
+session, `--log-dir DIR` writes a transcript per source, `--quiet` drops the progress spinner),
+`serve` (MCP stdio server), `search <query> [--tag T] [--limit N]`, `tags [tag]`,
+`lint [--stale-days N]`, `check [paths…]`, `view [--out PATH] [--no-open] [--obsidian]`.
+`citadel --version` prints the version and (like `--help`) needs no workspace.
+
+Tests (pytest, all offline — no CLI/network is ever spawned):
+
+```bash
+uv run pytest -q                                    # whole suite (~420 tests, ~3s)
+uv run pytest tests/test_ingest_core.py -q          # one file
+uv run pytest tests/test_ingest_core.py::test_ingest_creates_pages   # one test
+```
+
+New tests build on the shared fixtures in `tests/conftest.py` — that layer is THE pattern:
+`tmp_citadel` (a temp repo/wiki/raw/docs layout wired into `config.*`; `tmp_citadel_external`
+for the out-of-repo mounted-drive layout, `make_citadel` for custom ones), `seed_page` (write a
+canonical OKF page into the configured wiki), and `fake_agent` (a recording `FakeAgent`
+installed over `llm.run_ingest_session` — pages to write, an error to raise, or a
+`side_effect`). Don't re-create per-file `_wire*`/fake-session copies.
+
+Lint and format with **ruff** (config in `pyproject.toml`; CI gates both, alongside pytest):
+
+```bash
+uv run ruff check .       # lint
+uv run ruff format .      # auto-format (CI runs `ruff format --check .`)
+```
+
+Python 3.12+ is required. There is no separate build step — `pytest` and `ruff` are the checks.
+
+## Architecture
+
+**The `wiki/` directory _is_ the database.** No SQLite, no vector store, no second source of truth.
+Pages are markdown files with YAML frontmatter; everything (search, index, graph, provenance) is
+recomputed from them in memory.
+
+**Three layers** (the README and `citadel/rules/schema.md` are authoritative):
+1. `raw/` — immutable sources the agent reads but never edits.
+2. `wiki/` — the LLM-owned OKF bundle: pages routed *by kind* into `concepts/`, `objects/`,
+   `systems/`, `persons/`, `organizations/`, `projects/`, `abbreviations/`, `misc/` (see
+   `okf.folder_for_type`), cross-linked with relative markdown links, each fact carrying a footnote
+   citation.
+3. `citadel/rules/` — the schema/rules tree, packaged with the wheel (index:
+   `citadel/rules/README.md`; the repo-root `SCHEMA.md`/`AGENT_INGEST.md` are thin pointers):
+   `schema.md` (format contract) + `core.md` (agent behavior) are read every session, plus one
+   lifecycle brief from `tasks/`, any file-type brief from `formats/`, and the agent-judged
+   `genres/` briefs. These are **read by the ingest agent at run time** (referenced by path in the
+   prompt), so editing them changes how the wiki is built with **no code change**. Treat them as
+   part of the program.
+
+**Everything operates on a WORKSPACE**, not the repo checkout: a directory holding a
+`citadel.toml` marker (a pure marker, never config — scaffold one with `citadel init [DIR]`).
+Discovery order: `CITADEL_WORKSPACE` env var > nearest marker walking up from the CWD (nested
+markers shadow outer ones) > an env-dirs workspace (`CITADEL_WIKI_DIR`+`CITADEL_RAW_DIR` both
+set) > otherwise none: `config.WORKSPACE_FOUND` is False, `WORKSPACE_ROOT` falls back to the
+bare CWD, and every subcommand except `init` fails loud. The dev checkout carries a marker, so
+it is itself a workspace.
+
+**Ingest is the heart of the system** (`ingest.py` → `llm.py`). The flow per source:
+- `ingest.ingest()` partitions candidates into pending / already-ingested (sha match) / reorganized
+  (moved-or-duplicate) / unreadable (binary) / deleted (vanished from disk, full runs only).
+- For each pending source it runs the agent against a **per-source staging copy** of the wiki (a
+  sibling dir, never the live wiki), then snapshots before/after and **diffs by content hash** to
+  learn what the agent created/updated/deleted — the agent has no return value, its file edits *are*
+  the result.
+- It then re-imposes invariants on every changed page (`validate.validate_page` + `store.write_page`
+  to canonicalize YAML and stamp the timestamp), repairs renamed-page links, and **only on a fully
+  clean session promotes staging onto the live wiki** with a non-destructive copy-over-then-prune.
+  Any failure/timeout/Ctrl+C leaves the live wiki exactly as it was; the source is retried next run.
+  This all-or-nothing + network-share-hardened machinery (`_robust_*`, `robust_mkdir`) is load-bearing
+  — don't simplify it away.
+
+**`llm.py` is the ONLY place that talks to an LLM**, and it does so by shelling out to a CLI in
+agentic mode (`cwd` = workspace root, autonomous file tools). The prompt is **paths-only** — it references
+the source and rules by path, never embeds file content — which keeps argv tiny (the Windows
+`WinError 206` fix). `kind` selects the propagation: `ingest` (new), `reconcile` (changed source —
+update/remove stale facts, don't just append), `delete` (source removed — strip its provenance),
+`repo`/`repo-reconcile` (a whole git repo folded as one digest). `run_ingest_session` is the single
+seam tests monkeypatch.
+
+**Two checking layers, one implementation** (`validate.py`):
+- `citadel check` / `wiki_validate` — the **strict per-page gate** (required fields, honest/defined
+  citations, relative non-broken links, no `[[wikilinks]]`). The ingest agent self-runs it; ingest
+  re-runs it and fails the source on any error.
+- `citadel lint` (`lint.py`) — a **pure offline health check** (contradictions, orphans, missing
+  cites, broken links, stale, fabricated sources, undefined abbreviations). Only *structural*
+  problems (missing type, broken links, bad sources, wikilinks) flip its non-zero exit; the rest are
+  advisory. Both layers parse citations/links/fences through `grammar.py`, so lint and `citadel
+  check` agree by construction: a citation into `raw/` or `docs/` is legal provenance (never a
+  broken link), and a link inside a ``` code fence is literal text.
+
+**Other modules:** `okf.py` is the OKF format core (parse/dump, type→folder routing, link math, and
+the non-negotiable `safe_join` path guard — reuse it for any wiki-relative path). `grammar.py` is
+the **single home of the markdown grammar** (link/footnote/fence/Sources-heading parsing and the
+source-citation predicates) that `store`, `validate`, `lint`, and the viewer all parse through;
+never re-define any of it locally. `store.py` is the
+"database": `load()`, the single swappable `search()` seam, `rebuild_indexes()` (regenerates
+`index.md`, per-folder `index.md`, and `sources/index.md` mechanically from frontmatter +
+manifest), and the deterministic link-rewrite safety nets (`rewrite_links`, `rewrite_raw_references`,
+`find_raw_references`, `find_broken_links`), all fence-aware via `grammar.py`. `manifest.py` tracks idempotency in
+`wiki/.citadel_ingested.json` (per source: sha256 or git commit + importing model). `repo.py` builds
+the digest for git-repo sources. `extract.py` pulls text from Office files (stdlib-only); the legacy
+OLE/CFBF salvage lives in `extract_ole.py`, imported lazily only when a legacy `.ppt`/`.doc`/`.xls`
+is dispatched. `server.py` is the FastMCP stdio server (7 tools; only `wiki_ingest` mutates; tools
+never raise — they return error strings). The `viewer/` subpackage builds the self-contained offline
+HTML viewer (`template.html`/`app.css`/`app.js` are package-data assets loaded via `importlib.resources`). `config.py`
+resolves all paths/settings. `cli.py` mirrors the MCP tools as subcommands.
+
+## Conventions specific to this codebase
+
+- **`config.*` is read at call time** (`from . import config` then `config.WIKI_DIR`), never imported
+  by value — so tests can monkeypatch the whole filesystem layout. Honor this when adding code.
+- **Tests redirect everything to `tmp_path`** by monkeypatching `config.*` (including
+  `WORKSPACE_ROOT`, which the agent's `cwd` reads) and replace `llm.run_ingest_session` with a fake
+  that writes files into the temp wiki. No test spawns a real CLI. Follow that pattern; keep tests
+  offline.
+- **Never hand-edit generated files** — `index.md`, `log.md`, any `*/index.md`, `sources/index.md`,
+  `.citadel_viewer.html`, and `.citadel_ingested.json` are regenerated. The ingest agent prompt and
+  `store.delete_page` both refuse to touch them.
+- **Provenance grammar is load-bearing:** raw facts cite `[^sN]` → a real `raw/` file; model-supplied
+  facts use `[^llmN]` (source: `LLM`) and must never be disguised as raw citations. A `[^sN]` to a
+  missing file fails lint/check.
+- **`wiki/`, `raw/`, `docs/` can live outside the workspace** (e.g. a mounted network drive) via
+  `CITADEL_*_DIR`. Path handling distinguishes workspace-relative keys from absolute out-of-workspace
+  keys (`config.rel_or_abs_posix` / `source_path_for_key`) — preserve that when touching path logic.
+- **Cross-platform robustness is intentional**, not over-engineering: UTF-8 forcing, BOM stripping,
+  ASCII-only progress output, read-only-bit clearing, and network-share retry loops all fix real
+  Windows/SMB failures.
+- Config knobs live in the workspace-root `.env` (auto-loaded, gitignored; template:
+  `citadel/templates/env.example`): `CITADEL_LLM_CLI`,
+  `CITADEL_INGEST_MODEL`, `CITADEL_LLM_TIMEOUT`, `CITADEL_LLM_VERBOSE`, `CITADEL_LLM_LOG_DIR`,
+  `CITADEL_REPO_SUPPORT`, `CITADEL_WIKI_LANG` (target language of all wiki prose, default `en`),
+  `CITADEL_PDF_MODE` (`text` | `images`), `CITADEL_STYLE_PROFILES` (opt-in style capture, default
+  `0`), the `CITADEL_*_DIR` path overrides, and `*_CLI_PATH` binary overrides.

cite_citadel-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,180 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    name: test (${{ matrix.os }}, py${{ matrix.python-version }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        # 3.12 is the declared minimum (pyproject `requires-python`); 3.13/3.14 guard forward-compat.
+        # Full Python spread on ubuntu; windows/macos run only the edges (3.12 + 3.14) to keep the
+        # job count sane. Windows in particular is load-bearing: the SMB/network-share hardening
+        # (_robust_*, read-only-bit clearing, UTF-8 forcing) is exercised nowhere else in CI.
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: ["3.12", "3.13", "3.14"]
+        exclude:
+          - os: windows-latest
+            python-version: "3.13"
+          - os: macos-latest
+            python-version: "3.13"
+    defaults:
+      run:
+        # One shell on every OS (git-bash on Windows) so steps stay byte-identical across the matrix.
+        shell: bash
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+          enable-cache: true
+
+      - name: Install dependencies
+        run: uv sync --dev
+
+      - name: Run tests
+        run: uv run pytest -q
+
+      - name: Lint the bundled wiki (structural health check)
+        # The checkout root carries a citadel.toml marker, so the repo itself is a valid workspace.
+        run: uv run python -m citadel lint
+
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          python-version: "3.12"
+          enable-cache: true
+
+      - name: Install dependencies
+        run: uv sync --dev
+
+      - name: Ruff lint
+        run: uv run ruff check .
+
+      - name: Ruff format check
+        run: uv run ruff format --check .
+
+  # Regression gate for the phantom-workspace bug class: prove the WHEEL ALONE works — installed
+  # into a clean venv (no checkout on sys.path, no dev deps) and driven from a temp CWD outside
+  # the checkout (no citadel.toml above it, no CITADEL_* env). Each step checks exactly one layer,
+  # so the first red step names what broke:
+  #   uv build             -> build backend / project metadata
+  #   twine check          -> distribution metadata (PyPI would reject it)
+  #   venv + pip install   -> wheel installability (runtime deps mcp + pyyaml only)
+  #   packaged data        -> rules/ + templates/ actually shipped INSIDE the wheel
+  #   citadel --version    -> console entry point + version wiring
+  #   fail-loud guard      -> workspace discovery must NOT invent a phantom workspace
+  #   init / check / search -> workspace scaffold + read paths of the installed package
+  wheel-smoke:
+    name: wheel-smoke (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    defaults:
+      run:
+        # git-bash on Windows: globs, mktemp and `source` behave the same as on ubuntu.
+        shell: bash
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          python-version: "3.12"
+          enable-cache: true
+
+      - name: Build sdist + wheel
+        run: uv build
+
+      - name: Check distribution metadata (twine)
+        run: uvx twine check dist/*
+
+      - name: Install ONLY the wheel into a clean venv
+        # --seed puts pip into the venv; the activate glob resolves bin/ (POSIX) or Scripts/ (Windows).
+        # The venv's script dir goes onto $GITHUB_PATH once (pwd -W yields the native Windows path
+        # under git-bash; plain pwd elsewhere), so every later step runs citadel/python directly —
+        # from a smoke dir OUTSIDE the checkout, via per-step working-directory.
+        run: |
+          uv venv --seed smoke-venv
+          source smoke-venv/*/activate
+          python -m pip install --no-cache-dir dist/*.whl
+          echo "$(cd "$(dirname smoke-venv/*/activate)" && { pwd -W 2>/dev/null || pwd; })" >> "$GITHUB_PATH"
+          mkdir -p "${{ runner.temp }}/smoke"
+
+      - name: Wheel completeness — packaged rules + templates
+        # From a CWD with no ./citadel directory, imports MUST resolve to site-packages; the agent
+        # rules and the .env template are package data and must travel inside the wheel. The
+        # expected set is enumerated from the CHECKOUT (citadel/rules/**/*.md) — self-maintaining,
+        # so a newly added rules file that fails to ship in the wheel turns this step red.
+        working-directory: ${{ runner.temp }}/smoke
+        env:
+          CHECKOUT: ${{ github.workspace }}
+        run: |
+          python - <<'EOF'
+          import os
+          from importlib import resources
+          from pathlib import Path
+          import citadel
+          print("citadel imported from:", citadel.__file__)
+          src = Path(os.environ["CHECKOUT"]) / "citadel"
+          expected = sorted(p.relative_to(src).as_posix() for p in (src / "rules").rglob("*.md") if p.is_file())
+          if not expected:
+              raise SystemExit(f"no rules files found under {src / 'rules'} — wrong CHECKOUT path?")
+          expected.append("templates/env.example")
+          pkg = resources.files("citadel")
+          missing = [rel for rel in expected if not pkg.joinpath(rel).is_file()]
+          if missing:
+              raise SystemExit(f"wheel is missing packaged data: {missing} — check the hatchling wheel config / package layout")
+          print(f"packaged rules + templates present in the installed wheel ({len(expected)} files)")
+          EOF
+
+      - name: Smoke — citadel --version (console entry point)
+        working-directory: ${{ runner.temp }}/smoke
+        run: citadel --version
+
+      - name: Smoke — fail loud outside any workspace (no phantom workspace)
+        working-directory: ${{ runner.temp }}/smoke
+        run: |
+          if citadel check; then
+            echo "::error::'citadel check' succeeded in a CWD with no citadel.toml marker and no CITADEL_* dirs — workspace discovery invented a phantom workspace"
+            exit 1
+          fi
+          echo "OK: workspace-less 'citadel check' failed loud, as designed"
+
+      - name: Smoke — citadel init demo (workspace scaffold)
+        working-directory: ${{ runner.temp }}/smoke
+        run: |
+          citadel init demo
+          test -f demo/citadel.toml
+          test -f demo/.env
+          test -d demo/raw
+          test -d demo/wiki
+
+      - name: Smoke — citadel check on the fresh workspace (must be clean)
+        working-directory: ${{ runner.temp }}/smoke/demo
+        run: citadel check
+
+      - name: Smoke — citadel search on the empty wiki (must exit 0, no matches)
+        working-directory: ${{ runner.temp }}/smoke/demo
+        run: citadel search x