llm-wiki-compiler 0.7.0 → 0.9.0

package/README.md

@@ -6,10 +6,24 @@ Inspired by Karpathy's [LLM Wiki](https://gist.github.com/karpathy/442a6bf555914
 
 ![llmwiki demo](docs/images/demo.gif)
 
+## What you get
+
+- **Compiled wiki, not chunks.** A two-phase LLM pipeline turns raw sources into typed pages (`concept`, `entity`, `comparison`, `overview`) with paragraph- and claim-level citations back to source line ranges.
+- **Hybrid retrieval.** Semantic chunk embeddings (incremental, content-hash-aware) narrow hundreds of pages to a small top-K; BM25 reranking and wikilink-graph expansion build the final evidence pack.
+- **Local web viewer.** `llmwiki view` opens a read-only browser UI with sidebar navigation, search, a force-directed graph, and provenance/citation chips per page.
+- **Eval harness.** `llmwiki eval` measures health score (0–100), citation coverage and precision, optional LLM-as-judge support scoring, regression deltas, and CI-gateable thresholds.
+- **Source freshness.** `llmwiki lint` flags pages whose sources changed (`stale`) or were deleted (`orphaned`) since the last compile — surfaced across the viewer, MCP, context packs, the JSON export, and `llmwiki next` — and `llmwiki refresh --stale` repairs them with a targeted recompile.
+- **MCP server.** `llmwiki serve` exposes the full pipeline to Claude Desktop, Cursor, Claude Code, and any MCP-compatible agent — including `get_context_pack` for budgeted, citation-aware evidence packs.
+- **In-process SDK.** `createWiki({ root })` drives the whole pipeline programmatically — ingest, compile, query, status/freshness, context packs, export, eval — for embedding llmwiki in your own tooling without shelling out.
+- **Activity journal.** Every ingest, compile, and query appends a timestamped, machine-parseable entry to `log.md` — a human- and agent-readable audit trail of how the wiki was built, carrying page wikilinks and counts.
+- **Bridge to runtime memory.** `llmwiki export --target json --project-id <id>` produces a typed envelope that [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki) imports as one verbatim Atomic Memory record per page, preserving all advisory metadata.
+- **Provider-portable.** Anthropic, Claude Agent SDK (local Claude Code login, no API key), OpenAI-compatible (incl. local llama.cpp / vLLM), Ollama, GitHub Copilot.
+
 ## Who this is for
 
-- **AI researchers and engineers** building persistent knowledge from papers, docs, and notes
+- **AI researchers and engineers** building durable knowledge from papers, docs, and notes
 - **Technical writers** compiling scattered sources into a structured, interlinked reference
+- **Open-source maintainers** turning READMEs, ADRs, and design docs into a navigable knowledge base
 - **Anyone with too many bookmarks** who wants a wiki instead of a graveyard of tabs
 
 ## Quick start
@@ -22,12 +36,13 @@ export ANTHROPIC_API_KEY=sk-...
 # export LLMWIKI_PROVIDER=openai
 # export OPENAI_API_KEY=sk-...
 
-llmwiki ingest https://some-article.com
-llmwiki compile
+llmwiki quickstart ./notes.md
 llmwiki query "what is X?"
 llmwiki view --open
 ```
 
+`llmwiki quickstart ./notes.md` ingests one supported source, compiles the wiki, and opens the local viewer when pages are ready. Use `--no-open` to stop after compile, `--review` to queue candidates instead of writing pages, or `--json` for an agent-friendly envelope. If you're inside an existing project and unsure what to do next, run `llmwiki next`.
+
 
 <br>
 
@@ -77,6 +92,43 @@ Example with zero exports (Claude Code already configured):
 llmwiki compile
 ```
 
+### Claude Agent SDK (local Claude Code login)
+
+The `claude-agent` provider routes calls through the
+[Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-typescript)
+instead of the raw Messages API. It authenticates with your **local Claude Code
+login** (OAuth/subscription), so **no `ANTHROPIC_API_KEY` is required** — if you
+can run `claude` in your terminal, this provider works.
+
+> **Terms of use.** This provider drives your Claude Code / Agent SDK session
+> programmatically to compile wikis. That is not automatically appropriate for
+> every account type, plan, or environment. Before using it, review Anthropic's
+> current [Claude Code](https://www.anthropic.com/legal/consumer-terms) and
+> [Agent SDK](https://docs.anthropic.com/en/api/agent-sdk/overview) terms and
+> usage policies, and make sure your intended use complies with them.
+
+```bash
+export LLMWIKI_PROVIDER=claude-agent
+export LLMWIKI_MODEL=claude-sonnet-4-6  # optional; this is the default
+llmwiki compile
+```
+
+Notes:
+
+- Generation (`compile`) and structured extraction work off the local login with
+  no extra credentials.
+- Semantic search (`llmwiki query`) still needs embeddings, which Claude does not
+  provide. Set `VOYAGE_API_KEY` to enable them (same as the `anthropic`
+  provider); otherwise `query` falls back to lexical ranking.
+- To see what the SDK is doing, set `LLMWIKI_DEBUG=1` for a concise one-line trace
+  per SDK message (`[claude-agent] system:init`, `… assistant`, `… result:success`)
+  plus any `claude` subprocess errors. Use `LLMWIKI_DEBUG=verbose` to additionally
+  enable the SDK's full verbose logging.
+
+  ```bash
+  LLMWIKI_DEBUG=1 LLMWIKI_PROVIDER=claude-agent llmwiki compile
+  ```
+
 ### OpenAI-Compatible Local Servers
 
 Use the OpenAI provider for local OpenAI-compatible servers such as
@@ -179,28 +231,46 @@ A truncation warning prints to stderr when the cap fires so you know which conce
 <br>
 
 
-## Why not just RAG?
+## Why compile, not just retrieve?
 
-RAG retrieves chunks at query time. Every question re-discovers the same relationships from scratch. Nothing accumulates.
+llmwiki uses embeddings — chunk-level, incremental, with BM25 reranking. But the embedding layer sits **below** the compiled wiki, not in front of it.
 
-llmwiki **compiles** your sources into a wiki. Concepts get their own pages. Pages link to each other. When you ask a question with `--save`, the answer becomes a new page, and future queries use it as context. Your explorations compound.
+**RAG retrieves chunks at query time.** Every question re-discovers the same relationships from scratch. The wiki structure, citation graph, and merged-concept disambiguation never accumulate; they get re-invented per query.
 
-This is complementary to RAG, not a replacement. RAG is great for ad-hoc retrieval over large corpora. llmwiki gives you a persistent, structured artifact to retrieve from.
+**llmwiki compiles your sources into a wiki first.** Concepts get their own typed pages. Concepts shared across multiple sources are merged into one page instead of competing as duplicate chunks. Pages link to each other via `[[wikilinks]]`. When you ask a question with `--save`, the answer becomes a new page, and future queries use it as context.
+
+Then semantic retrieval, BM25 reranking, and graph expansion run over the compiled artifact — narrowing hundreds of pages to a tight, citation-traceable evidence pack.
 
 ```
 RAG:     query → search chunks → answer → forget
-llmwiki: sources → compile → wiki → query → save → richer wiki → better answers
+llmwiki: sources → compile → wiki → embed → query → save → richer wiki → better answers
 ```
 
+llmwiki is complementary to traditional RAG: use RAG for ad-hoc retrieval over noisy or fast-changing corpora; use llmwiki when you want a persistent, structured, citation-traceable artifact that compounds.
+
 ## How it works
 
 ```
-sources/  →  SHA-256 hash check  →  LLM concept extraction  →  wiki page generation  →  [[wikilink]] resolution  →  index.md
+sources/  →  hash check  →  LLM concept extraction  →  page generation  →  [[wikilink]] resolve
+              │                                                             ↓
+              │                                        chunk embeddings  ←  wiki/  →  index.md
+              │                                               ↓
+              │                             semantic search + BM25 rerank + graph expansion
+              │                                               ↓
+              │                                    llmwiki query / context / MCP
+              ↓
+   stale / orphaned pages  →  llmwiki refresh --stale  →  recompile changed owners, clean up orphans
 ```
 
-**Two-phase pipeline.** Phase 1 extracts all concepts from all sources. Phase 2 generates pages. This eliminates order-dependence, catches failures before writing anything, and merges concepts shared across multiple sources into single pages.
+**Two-phase compile.** Phase 1 extracts all concepts from every source; Phase 2 generates pages. Splitting the phases eliminates order-dependence, catches extraction failures before anything is written, merges concepts shared across multiple sources into a single page, and marks pages whose sources were all deleted as `orphaned` rather than silently dropping them.
 
-**Incremental.** Only changed sources go through the LLM. Everything else is skipped via hash-based change detection.
+**Incremental everywhere.** Hash-based change detection on sources, content-hash-aware embedding updates, and cached citation judgements mean only changed work runs through the LLM. Recompiling after editing one source touches just the pages that source contributed to.
+
+**Source freshness and repair.** Every page records the sources — and their content hashes — that produced it. On any later command, llmwiki compares those recorded hashes against `sources/` on disk: a page whose sources changed since the last compile is `stale`, and a page whose sources were all deleted is `orphaned`. `llmwiki lint`, `llmwiki status`, the viewer, the JSON export, and the MCP tools surface this without recompiling anything. `llmwiki refresh --stale` then repairs it — recompiling only the changed sources that own stale pages and cleaning up orphaned ones, while deliberately leaving unrelated new sources for a full `llmwiki compile`. `--dry-run` previews the plan with no LLM calls.
+
+**Hybrid retrieval.** `.llmwiki/embeddings.json` v2 carries page- and chunk-level vectors. `llmwiki query` and `llmwiki context` narrow hundreds of pages down to a chunk-level top-K via cosine similarity, then rerank with BM25 and expand along the wikilink graph for the final evidence pack.
+
+**Citation-traceable.** Paragraphs carry `^[source.md]` markers; specific claims pin to `^[source.md:42-58]` line ranges. `llmwiki lint` validates that every citation resolves to a real file and line range; `llmwiki eval` measures citation precision and (optionally) LLM-judged claim support.
 
 **Compounding queries.** `llmwiki query --save` writes the answer as a wiki page and immediately rebuilds the index. Saved answers show up in future queries as context.
 
@@ -247,26 +317,43 @@ Pages include source attribution in frontmatter. Paragraphs are annotated with `
 |---------|-------------|
 | `llmwiki ingest <url\|file>` | Fetch a URL or copy a local file into `sources/` |
 | `llmwiki ingest-session <path>` | Import a Claude/Codex/Cursor session export (single file or whole directory) into `sources/` |
+| `llmwiki quickstart <source>` | Ingest a source and compile a wiki in one step; supports `--review`, `--no-open`, `--provider`, `--lang`, and `--json` |
 | `llmwiki compile` | Incremental compile: extract concepts, generate wiki pages |
 | `llmwiki compile --review` | Write candidate pages to `.llmwiki/candidates/` instead of `wiki/` so you can review before they land |
 | `llmwiki compile --lang <code>` | Generate wiki content in the given language (e.g. `Chinese`, `ja`, `zh-CN`); also works on `query` |
+| `llmwiki refresh --stale [--dry-run]` | Repair stale/orphaned pages: recompile the sources that own stale pages and clean up deleted owners, skipping unrelated new sources. `--dry-run` previews the plan with no LLM calls or writes |
 | `llmwiki review list` | List pending candidate pages |
 | `llmwiki review show <id>` | Print a candidate's title, summary, and body |
 | `llmwiki review approve <id>` | Promote a candidate into `wiki/` and refresh index/MOC/embeddings |
 | `llmwiki review reject <id>` | Archive a candidate without touching `wiki/` |
+| `llmwiki rules extract` | Extract machine-actionable rule candidates from changed sources into `.llmwiki/rule-candidates/` |
+| `llmwiki rules list` | List pending rule candidates |
+| `llmwiki rules approve <id>` / `reject <id>` | Approve or reject a rule candidate |
+| `llmwiki rules export` | Emit approved rule candidates as a JSON array for a downstream rule importer |
 | `llmwiki schema init` | Write a starter `.llmwiki/schema.json` file |
 | `llmwiki schema show` | Print the resolved schema for the current project |
 | `llmwiki query "question"` | Ask questions against your compiled wiki |
 | `llmwiki query "question" --save` | Answer and save the result as a wiki page |
-| `llmwiki export [--target <name>]` | Export the wiki to portable formats — `llms.txt`, `llms-full.txt`, JSON, JSON-LD, GraphML, Marp slides |
+| `llmwiki export [--target <name>] [--project-id <id>]` | Export the wiki to portable formats — `llms.txt`, `llms-full.txt`, JSON, JSON-LD, GraphML, Marp slides. `--project-id` pins a stable identifier inside the JSON envelope so downstream importers (e.g. [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki)) can derive deterministic external IDs |
 | `llmwiki view [--open]` | Start a read-only local web viewer for browsing, searching, and inspecting the compiled wiki |
-| `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, low confidence, contradictions, etc.) |
+| `llmwiki next [--json]` | Show the recommended next action for this project (read-only); `--json` emits a stable envelope for agents |
+| `llmwiki context "<prompt>" [--json]` | Build an agent-ready evidence pack (primary pages, citations, neighbors, suggested actions) — same v1 envelope as MCP `get_context_pack` |
+| `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, low confidence, contradictions, stale pages whose sources changed, etc.) |
+| `llmwiki eval [--suite fast\|full]` | Measure wiki quality: health score (0–100), citation coverage, corpus stats. `--suite full` adds LLM-as-judge citation support scoring |
+| `llmwiki eval cache show` | Print score distribution and top-cited pages from the citation judgement cache |
+| `llmwiki eval cache clear` | Remove the citation judgement cache |
+| `llmwiki eval report` | Print the most recent eval report |
+| `llmwiki eval history [--n N]` | Show a trend table of past eval runs from `history.jsonl` |
+| `llmwiki eval judgements [--score 0\|1\|2] [--page slug]` | Inspect individual citation judgements with optional score or page filters |
 | `llmwiki watch` | Auto-recompile when `sources/` changes |
 | `llmwiki serve [--root <dir>]` | Start an MCP server exposing wiki tools to AI agents |
 
+`llmwiki context --include-sources` and MCP `get_context_pack` with `includeSources: true` are opt-in because they can return raw snippets from files under `sources/`. Path confinement prevents reads outside `sources/`, but only enable source windows for agents you trust with the ingested source text.
+
 ## Output
 
 ```
+log.md              append-only activity journal (ingests, compiles, queries)
 wiki/
   concepts/         one .md file per concept, with YAML frontmatter
   queries/          saved query answers, included in index and retrieval
@@ -277,7 +364,31 @@ wiki/
   candidates/archive/  rejected candidates kept for audit
 ```
 
-Obsidian-compatible. `[[wikilinks]]` resolve to concept titles.
+Obsidian-compatible. `[[wikilinks]]` resolve to concept titles — or to any page that declares the term in its `aliases` frontmatter, so links survive renames and synonyms.
+
+`log.md` records what happened and when. Each entry is a heading with a fixed
+prefix — `## [YYYY-MM-DDThh:mm:ssZ] operation | description` (an ISO 8601 UTC
+timestamp) — followed by a short bullet body carrying page wikilinks and counts:
+
+```markdown
+## [2026-06-05T09:14:02Z] ingest | Attention Is All You Need
+- Source: https://arxiv.org/abs/1706.03762
+- Saved: sources/attention-is-all-you-need.md
+- Chars: 38,214
+
+## [2026-06-05T09:15:30Z] compile | 1 source(s) → 6 page(s)
+- Sources: attention-is-all-you-need.md
+- Created: [[self-attention]], [[multi-head-attention]], [[transformer]]
+- Updated: [[positional-encoding]]
+
+## [2026-06-05T09:16:11Z] query | What is multi-head attention?
+- Pages: [[multi-head-attention]], [[self-attention]]
+```
+
+Only headings start with `## [`, so the gist's recipe still works even with the
+bodies: `grep "^## \[" log.md | tail -5` shows the five most recent operations.
+Where `index.md` organizes content for discovery, `log.md` tracks temporal
+progression.
 
 ## Local web viewer
 
@@ -369,6 +480,55 @@ The schema supports four page kinds:
 
 Schema rules can set per-kind `minWikilinks` and optional `seedPages`. Compile can materialize seed pages such as overviews, lint enforces page-kind-specific cross-link minimums, and review candidates surface schema violations before approval.
 
+## Eval / quality measurement
+
+`llmwiki eval` gives the wiki a quantitative health score and tracks citation quality over time, making it possible to detect regressions after a recompile.
+
+```bash
+llmwiki eval                   # fast suite: health score, citation coverage, corpus stats
+llmwiki eval --suite full      # + LLM-as-judge citation support scoring (requires API)
+llmwiki eval report            # re-print the most recent report
+llmwiki eval history           # trend table across past runs
+llmwiki eval history --n 10    # limit to last 10 entries
+llmwiki eval judgements        # all cached citation judgements
+llmwiki eval judgements --score 0          # only unsupported citations
+llmwiki eval judgements --page some-slug   # filter to one page
+llmwiki eval cache show        # score distribution + top-cited pages
+llmwiki eval cache clear       # wipe the citation judgement cache
+```
+
+**What it measures:**
+
+- **Health score (0–100)** aggregates all lint rules. Errors (broken citations, broken wikilinks, duplicate concepts) cost more than warnings.
+- **Citation coverage** — fraction of prose paragraphs that carry a `^[...]` marker, plus citation precision (fraction of citations pointing to existing source files).
+- **Citation support (full suite)** — samples up to N `(claim, source span)` pairs, asks a judge model to score each 0–2 (unsupported → fully supported), and caches results so subsequent runs only re-judge new pairs.
+- **Source utilization & citation depth** — the fraction of a page's valid sources that are actually cited (`source_utilization_rate`), and the share of citations pinned to specific line ranges rather than whole files (`claim_level_citation_rate`). Source warnings flag sources excluded from compilation (e.g. out-of-tree symlinks), gateable via `source_warnings_max`.
+- **Corpus stats** — page count, source count, total wiki characters, embedding counts, appended to `history.jsonl` for trend tracking.
+- **Regression deltas** — current report is diffed against the previous entry in history.
+
+**CI thresholds:** add `.llmwiki/eval/thresholds.yaml` to configure minimum acceptable scores:
+
+```yaml
+health_score: 85
+citation_coverage_percent: 70
+citation_precision_percent: 90
+citation_support_mean: 1.4   # only checked when --suite full
+source_utilization_rate: 0.9 # min fraction of valid sources cited by a page
+source_warnings_max: 0       # max excluded sources (out-of-tree symlinks, etc.)
+claim_level_citation_rate: 0.5 # min fraction of citations with line ranges
+```
+
+Threshold violations are listed in the report. Exit code is non-zero when any threshold is breached, suitable for CI gating.
+
+**Artifacts** written under `.llmwiki/eval/`:
+
+```
+.llmwiki/eval/
+  history.jsonl          one JSON line per eval run
+  citation-cache.jsonl   one JSON line per citation judgement
+  thresholds.yaml        optional CI threshold config
+```
+
 </details>
 
 
@@ -385,10 +545,8 @@ Try it on any article or document:
 
 ```bash
 mkdir my-wiki && cd my-wiki
-llmwiki ingest https://en.wikipedia.org/wiki/Andrej_Karpathy
-llmwiki compile
+llmwiki quickstart https://en.wikipedia.org/wiki/Andrej_Karpathy
 llmwiki query "What terms did Andrej coin?"
-llmwiki view --open
 ```
 
 See `examples/basic/` in the repo for pre-generated output you can browse without an API key.
@@ -437,7 +595,7 @@ Add to your client's MCP config (e.g. `claude_desktop_config.json`):
 }
 ```
 
-Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for a configured provider on each call. Read-only tools (`read_page`, `lint_wiki`, `wiki_status`) and `ingest_source` work without any credentials.
+Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for a configured provider on each call. Read-only tools (`read_page`, `lint_wiki`, `wiki_status`) and `ingest_source` work without any credentials. `get_context_pack` is read-only and provider credentials are optional — when present, semantic retrieval is used; otherwise the tool falls back to lexical ranking and surfaces an `embedding-store-missing` or `query-embedding-unavailable` warning.
 
 ### Tools
 
@@ -449,7 +607,9 @@ Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for
 | `search_pages` | Return full content of pages relevant to a question. |
 | `read_page` | Read a single page by slug (concepts/ then queries/). |
 | `lint_wiki` | Run quality checks; returns structured diagnostics. |
-| `wiki_status` | Page count, source count, orphans, pending changes (read-only). |
+| `wiki_status` | Page/source counts, stale and orphaned pages, a `stateStatus` field, and pending changes (read-only). |
+| `get_context_pack` | Build an agent-ready evidence pack (primary pages, semantic chunks, graph neighbors, citations, per-page freshness, warnings, suggested actions) — same v1 JSON envelope as `llmwiki context --json`. `get_context_pack` **packages evidence**; `query_wiki` **generates answers**. |
+| `run_eval` | Score wiki quality (the fast suite needs no API key; the full suite LLM-judges a sample of citations); read-only. |
 
 ### Resources
 
@@ -460,6 +620,8 @@ Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for
 | `llmwiki://query/{slug}` | A single saved query page. |
 | `llmwiki://sources` | List of ingested source files with metadata. |
 | `llmwiki://state` | Compilation state (per-source hashes, last compile times). |
+| `llmwiki://eval/report` | The most recent eval report. |
+| `llmwiki://eval/history` | Trend of past eval runs. |
 
 </details>
 
@@ -471,31 +633,115 @@ Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for
 <br>
 
 
-## Limitations
+<details>
+<summary><span style="font-size: 1.4em;"><strong>SDK — programmatic API — click to expand</strong></span></summary>
+
+
+## SDK — `createWiki()`
+
+Drive llmwiki in-process instead of shelling out to the CLI. `createWiki({ root })` returns a `Wiki` facade bound to a project directory. Every method runs silently (no console output) and is concurrency-safe — quiet mode is scoped per async call, not via a global flag. `createWiki` is exported from the package entry, so `import { createWiki } from "llm-wiki-compiler"` works for any installed version.
 
-Early software. Best for small, high-signal corpora (a few dozen sources). Query routing is index-based.
+```ts
+import { createWiki } from "llm-wiki-compiler";
 
-**Honest about truncation.** Sources that exceed the character limit are truncated on ingest with `truncated: true` and the original character count recorded in frontmatter, so downstream consumers know they're working with partial content.
+const wiki = createWiki({ root: "./my-wiki" });
+
+await wiki.ingestText({ title: "Notes", text: "Raw text to compile…" });
+await wiki.compile();                        // needs LLM credentials
+const { answer } = await wiki.query("What did I note about X?");
+const status = await wiki.status();          // no credentials needed
+```
+
+### Methods
+
+| Method | What it does | LLM creds |
+|--------|--------------|:---------:|
+| `ingest({ source })` | Fetch a URL or read a local file into `sources/`. **Trusted input only** — a server-side fetch + local-file-read primitive (SSRF risk); use `ingestText` for untrusted content. | No |
+| `ingestText({ title, text })` | Ingest raw text — the safe path for untrusted content (no fetch, no file read). | No |
+| `compile(options?)` | Compile pending sources into wiki pages. `options.review` queues candidates instead of writing. **Sends source content to the provider.** | Yes |
+| `search(question)` | Retrieve and hydrate the most relevant page records. | Yes |
+| `query(question, options?)` | Grounded answer; `options.save` persists it as a page, `options.debug` returns retrieval detail. | Yes |
+| `getPage(ref)` / `listPages(options?)` | Read one page / list pages with filters and cursor pagination. | No |
+| `listSources(options?)` / `getSource(id)` / `deleteSource(id)` | List, read, or delete ingested sources. `id` is the `IngestResult.filename` (e.g. `"note.md"`); `deleteSource` reconciles the compiled page on the next `compile()`. | No |
+| `status()` | Read-only status snapshot — counts, freshness, pending changes. | No |
+| `lint()` | Run all lint rules; severity-counted summary. | No |
+| `getContextPack({ prompt, budget?, depth?, topPages?, topChunks? })` | Build a v1 context pack — same envelope as MCP `get_context_pack`. Semantic retrieval when embeddings exist, lexical fallback otherwise. | No |
+| `exportJson(options?)` | Structured JSON export document (same shape as `llmwiki export --target json`). | No |
+| `runEval({ mode, record? })` | Eval harness. `mode: "fast"` is credential-free; `"full"` LLM-judges a sample of citations. | full only |
+
+**Notes.** Methods that need a provider throw `ProviderUnavailableError` when no credentials are configured; the rest run credential-free. Output is suppressed and there is no progress callback in v1 (`compile`/`runEval` on a large corpus can run for minutes with no feedback). `status()`, `lint()`, and `exportJson()` each hash the full source corpus per call (no cross-call cache) — avoid calling them in a hot loop. All result types (`CompileResult`, `QueryResult`, `WikiStatus`, `ContextPack`, …) are exported from the package for typed consumption.
+
+</details>
+
+
+<br>
+
+---
+
+<br>
+
+
+## Companion: Atomic Memory
+
+llmwiki and [Atomic Memory](https://github.com/atomicstrata/atomicmemory) are complementary layers of open context infrastructure, both maintained by [Atomic Strata](https://github.com/atomicstrata):
+
+- **llmwiki** gives you a persistent **knowledge base** — durable markdown compiled from your sources, inspectable on disk.
+- **Atomic Memory** gives your agents persistent **working memory** — runtime context that's searchable, correctable, scoped, and inspectable over time.
+
+Use them independently or together. Each remains valuable on its own — llmwiki as a notebook, RAG index, CI-checked knowledge base, or domain pack source; Atomic Memory as a runtime memory layer for any agent or app.
+
+The [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki) bridge ingests `llmwiki export --target json --project-id <id>` envelopes as one verbatim Atomic Memory record per wiki page, preserving all advisory metadata (kind, citations, confidence, provenance state, contradictions, aliases, freshness) under `memory.metadata.llmwiki.*`. See the [bridge cookbook](https://github.com/atomicstrata/atomicmemory/blob/main/packages/llmwiki/docs/cookbook.md) for the full compile → export → import → package workflow.
+
+## Scale and what works
+
+Still early software, but the scale story has matured well past the "few dozen sources" era.
+
+- **Semantic chunk retrieval** (`.llmwiki/embeddings.json` v2) narrows hundreds of pages down to a small top-K before LLM selection, with BM25 reranking and graph-neighborhood expansion layered on top.
+- **Incremental everything.** Hash-based source-change detection, content-hash-aware embedding updates, cached citation judgements. Re-running on an unchanged corpus is a few seconds.
+- **Lexical fallback.** Index-based routing kicks in automatically when no embedding store is present or the active provider has no embedding credentials, surfacing a stable warning code rather than hard-failing.
+
+**Honest about truncation.** Sources that exceed the character limit are truncated on ingest with `truncated: true` and the original character count recorded in frontmatter, so downstream consumers know they're working with partial content. A per-concept prompt budget prevents popular shared concepts from crashing compile.
+
+**Where it's still early.** No source-freshness watchdog yet (re-ingest detects content changes, but doesn't proactively re-check URLs). No team / multi-writer conflict resolution. The viewer is read-only by design — write operations go through the CLI or MCP.
 
 ## Karpathy's LLM Wiki pattern vs this compiler
 
-Karpathy describes an abstract pattern for turning raw data into compiled knowledge. Here's how llmwiki maps to it:
+Karpathy described an abstract pattern for turning raw data into compiled knowledge. Here's how llmwiki maps to it today:
 
 | Karpathy's concept | llmwiki | Status |
 |---|---|---|
-| Data ingest | `llmwiki ingest` | Implemented |
-| Compile wiki | `llmwiki compile` | Implemented |
-| Q&A | `llmwiki query` | Implemented |
+| Data ingest | `llmwiki ingest`, `ingest-session` (Claude/Codex/Cursor) | Implemented |
+| Compile wiki | `llmwiki compile` (two-phase, incremental) | Implemented |
+| Q&A | `llmwiki query` (semantic + BM25 + graph expansion) | Implemented |
 | Output filing (save answers back) | `llmwiki query --save` | Implemented |
 | Auto-recompile | `llmwiki watch` | Implemented |
-| Linting / health-check pass | `llmwiki lint` | Implemented |
-| Agent integration | `llmwiki serve` (MCP server) | Implemented |
-| Image support | `llmwiki ingest <image>` | Implemented |
+| Linting / health-check pass | `llmwiki lint` + `llmwiki eval` (CI-gateable) | Implemented |
+| Agent integration | `llmwiki serve` MCP server with `get_context_pack` | Implemented |
+| Multimodal ingest | Images, PDFs, transcripts via `llmwiki ingest` | Implemented |
 | Marp slides | `llmwiki export --target marp` | Implemented |
+| Bridge to runtime memory | `llmwiki export --target json --project-id` → [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki) | Implemented |
 | Fine-tuning | — | Not yet implemented |
 
 ## Roadmap
 
+Shipped in 0.9.0:
+
+- ✅ Source freshness — `llmwiki lint` flags pages whose sources changed (`stale`) or were all deleted (`orphaned`) since compile, surfaced across MCP (`wiki_status`, `get_context_pack`), context packs, the viewer (badges, a per-axis filter, health counts, a corrupt-state banner), the JSON export, and `llmwiki next`
+- ✅ `llmwiki refresh --stale` — repairs stale/orphaned pages with a targeted recompile of their changed owning sources (and deleted-owner cleanup), skipping unrelated new sources; `--dry-run` previews with no LLM calls or writes
+- ✅ JSON export bridge contract — `llmwiki export --target json --project-id <id>` adds per-page `path`, `kind`, advisory confidence/provenance, flattened citations, aliases, and freshness so downstream importers (e.g. [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki)) can ingest pages as durable memory records
+- ✅ In-process SDK — `createWiki()` exposes the compiler in-process, with source-backed write APIs for programmatic callers
+- ✅ Eval over MCP + richer metrics — `run_eval` tool and read-only `llmwiki://eval/report`/`llmwiki://eval/history` resources, plus source-utilization and citation-depth metrics with a `source_warnings_max` CI gate
+- ✅ Rule-candidate extraction — extract reusable rule candidates from sources with review/approve and a JSON export pipeline
+- ✅ Claude Agent provider — authenticates through a local Claude Code login (bundled plan tokens, no separate API key)
+- ✅ Alias-aware wikilinks — the viewer resolves a `[[term]]` link to any page that declares `term` in its `aliases` frontmatter, not just an exact slug match
+
+Shipped in 0.8.0:
+
+- ✅ Guided project flow — `llmwiki next` recommends the next useful command, and `llmwiki quickstart <source>` ingests, compiles, and opens the viewer in one step
+- ✅ Graph/context layer — `llmwiki context` and MCP `get_context_pack` produce token-budgeted evidence packs with primary pages, graph neighbors, citations, optional source windows, warnings, and suggested actions
+- ✅ Viewer graph route — `llmwiki view` includes a force-directed `#/graph` route for exploring page relationships
+- ✅ Evaluation harness — `llmwiki eval` measures health score, citation coverage/precision, corpus stats, regression deltas, optional LLM-as-judge citation support, and CI thresholds
+
 Shipped in 0.7.0:
 
 - ✅ Read-only local web viewer — `llmwiki view` with sidebar navigation, markdown rendering, search, metadata, health counts, and provenance/citation chips
@@ -536,11 +782,10 @@ Shipped in 0.2.0:
 
 Next up:
 
-- **Graph/context layer** — page-neighborhood tools, graph paths, gap detection, and token-budgeted context packs for agents.
-- **Evaluation harness** — benchmark answer quality, citation accuracy, update drift, retrieval recall, and scale curves against serious retrieval baselines.
 - **Task and decision ledger** — turn session ingest into durable agent memory: goals, decisions, open questions, outcomes, and next-agent handoffs.
 - **Rollback, audit, and source lifecycle** — undo/reverse ingest, compile diff reports, stale-claim checks, freshness reports, and a durable operation log.
 - **Domain templates** — schema/prompt packs for research, codebase docs, team handbooks, decision logs, and standards/regulations.
+- **Eval extensions** — retrieval recall suites, update-drift benchmarks, and comparisons against serious retrieval baselines.
 
 Later / open to discussion:
 
@@ -549,7 +794,7 @@ Later / open to discussion:
 - Codex OAuth provider — ChatGPT subscription auth as a dedicated provider, with clear token refresh and embedding-limit behavior
 - Team-chat connectors for Slack/Discord/Teams-style institutional memory
 
-If you like ambitious problems: **local web UI**, **graph/context packs**, and **eval harness** are the meatiest next contributions. Open an issue to claim one or kick off a design discussion.
+If you like ambitious problems: **task/decision ledger**, **rollback/audit tooling**, and **eval extensions** are the meatiest next contributions. Open an issue to claim one or kick off a design discussion.
 
 Explicitly not planned (good ideas, just not for this repo): full static-site generator, desktop or mobile apps, fine-tuning, a formal ontology engine, heavy graph database infrastructure.
 
@@ -557,6 +802,10 @@ Explicitly not planned (good ideas, just not for this repo): full static-site ge
 
 Node.js >= 24, plus provider credentials (for Anthropic: `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`).
 
+## About
+
+llmwiki is maintained by [Atomic Strata](https://github.com/atomicstrata), the team behind [Atomic Memory](https://github.com/atomicstrata/atomicmemory). Atomic Strata builds open context infrastructure: durable compiled knowledge with llmwiki, runtime memory with Atomic Memory.
+
 ## License
 
 MIT