llm-wiki-compiler 0.8.0 → 0.9.0

package/CHANGELOG.md

@@ -5,6 +5,36 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] - 2026-06-08
+
+Adds an end-to-end source-freshness loop — detect stale pages, surface them everywhere, and repair them with a targeted recompile — plus an in-process SDK with source-backed write APIs, a JSON export bridge contract for downstream importers, richer eval metrics, rule-candidate extraction, and a local-login Claude Agent provider.
+
+### Added
+
+- **Source freshness** — `llmwiki lint` flags pages whose sources changed (`stale`) or were all deleted (`orphaned`) since the last compile, computed on demand from `.llmwiki/state.json` and the current `sources/`. Freshness is surfaced across MCP (`wiki_status` stale/orphaned lists and a `stateStatus` field, plus `get_context_pack`), context packs (per-page `freshnessStatus`/`contradicted`/`archived` and a `stale-page` warning), the local viewer (STALE/ORPHANED/CONTRADICTED/ARCHIVED badges, a per-axis filter, health-pane counts, and a corrupt-state banner), the JSON export, and `llmwiki next`.
+- **`llmwiki refresh --stale [--dry-run]`** — a targeted recompile that repairs stale/orphaned pages by recompiling their changed owning sources and cleaning up deleted owners, while deliberately skipping unrelated new sources. `--dry-run` previews the plan with no LLM calls and no writes; cleanup-only refreshes require no API key.
+- **JSON export bridge contract** — `llmwiki export --target json --project-id <id>` adds per-page `path`, `kind`, advisory confidence/provenance, flattened citations, aliases, and freshness so downstream importers (e.g. [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki)) can ingest pages as durable memory records.
+- **Eval over MCP** — a `run_eval` MCP tool (the fast suite needs no API key; the full suite LLM-judges a sample of citations), plus read-only `llmwiki://eval/report` and `llmwiki://eval/history` resources.
+- **Eval source-utilization metrics** — source-utilization and citation-depth dimensions, surfaced source warnings, a frame-safe report, and a `source_warnings_max` CI gate.
+- **Rule-candidate extraction** — extract reusable rule candidates from sources with review/approve and a JSON export pipeline.
+- **In-process SDK** — `createWiki()` exposes the compiler in-process, with source-backed write APIs (`writeStatus`, `listSources`/`getSource`/`deleteSource`) for programmatic callers.
+- **Claude Agent SDK provider** — a provider that authenticates through a local Claude Code login and uses bundled plan tokens, so no separate API key is required.
+- **Alias-aware wikilinks** — the viewer resolves a `[[term]]` link to any page that declares `term` in its `aliases` frontmatter, not just an exact slug match.
+- **Append-only activity journal** — an append-only `log.md` records ingest, compile, review, and export activity.
+
+### Changed
+
+- Upgraded core dependencies — zod 3 → 4, openai 4 → 6, and `@anthropic-ai/sdk` 0.39 → 0.101 — and bumped the default model to `claude-sonnet-4-6` (the previous default was deprecated).
+
+### Fixed
+
+- Three read-only paths (`wiki_status`, the `llmwiki://state` MCP resource, and the viewer startup snapshot) no longer write a `.bak` file when `.llmwiki/state.json` is corrupt; corrupt and missing state are now surfaced explicitly instead of being swallowed.
+- `wiki_status` derives pending source changes from the freshness snapshot instead of running a redundant second source-hash pass.
+
+### Contributors
+
+Thanks to **@alvins82** for the Claude Agent SDK provider (#81) and the append-only activity journal (#85), **@dohu012** for source-utilization and citation-depth eval metrics (#86), and **@joshuaknipe** for the `run_eval` MCP tool and eval resources (#74).
+
 ## [0.8.0] - 2026-05-26
 
 Adds guided project next steps, one-command quickstart, agent-ready context graph packs, a viewer graph route, and the first eval harness for measuring wiki quality over time.

package/README.md

@@ -6,10 +6,24 @@ Inspired by Karpathy's [LLM Wiki](https://gist.github.com/karpathy/442a6bf555914
 
 ![llmwiki demo](docs/images/demo.gif)
 
+## What you get
+
+- **Compiled wiki, not chunks.** A two-phase LLM pipeline turns raw sources into typed pages (`concept`, `entity`, `comparison`, `overview`) with paragraph- and claim-level citations back to source line ranges.
+- **Hybrid retrieval.** Semantic chunk embeddings (incremental, content-hash-aware) narrow hundreds of pages to a small top-K; BM25 reranking and wikilink-graph expansion build the final evidence pack.
+- **Local web viewer.** `llmwiki view` opens a read-only browser UI with sidebar navigation, search, a force-directed graph, and provenance/citation chips per page.
+- **Eval harness.** `llmwiki eval` measures health score (0–100), citation coverage and precision, optional LLM-as-judge support scoring, regression deltas, and CI-gateable thresholds.
+- **Source freshness.** `llmwiki lint` flags pages whose sources changed (`stale`) or were deleted (`orphaned`) since the last compile — surfaced across the viewer, MCP, context packs, the JSON export, and `llmwiki next` — and `llmwiki refresh --stale` repairs them with a targeted recompile.
+- **MCP server.** `llmwiki serve` exposes the full pipeline to Claude Desktop, Cursor, Claude Code, and any MCP-compatible agent — including `get_context_pack` for budgeted, citation-aware evidence packs.
+- **In-process SDK.** `createWiki({ root })` drives the whole pipeline programmatically — ingest, compile, query, status/freshness, context packs, export, eval — for embedding llmwiki in your own tooling without shelling out.
+- **Activity journal.** Every ingest, compile, and query appends a timestamped, machine-parseable entry to `log.md` — a human- and agent-readable audit trail of how the wiki was built, carrying page wikilinks and counts.
+- **Bridge to runtime memory.** `llmwiki export --target json --project-id <id>` produces a typed envelope that [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki) imports as one verbatim Atomic Memory record per page, preserving all advisory metadata.
+- **Provider-portable.** Anthropic, Claude Agent SDK (local Claude Code login, no API key), OpenAI-compatible (incl. local llama.cpp / vLLM), Ollama, GitHub Copilot.
+
 ## Who this is for
 
-- **AI researchers and engineers** building persistent knowledge from papers, docs, and notes
+- **AI researchers and engineers** building durable knowledge from papers, docs, and notes
 - **Technical writers** compiling scattered sources into a structured, interlinked reference
+- **Open-source maintainers** turning READMEs, ADRs, and design docs into a navigable knowledge base
 - **Anyone with too many bookmarks** who wants a wiki instead of a graveyard of tabs
 
 ## Quick start
@@ -78,6 +92,43 @@ Example with zero exports (Claude Code already configured):
 llmwiki compile
 ```
 
+### Claude Agent SDK (local Claude Code login)
+
+The `claude-agent` provider routes calls through the
+[Claude Agent SDK](https://github.com/anthropics/claude-agent-sdk-typescript)
+instead of the raw Messages API. It authenticates with your **local Claude Code
+login** (OAuth/subscription), so **no `ANTHROPIC_API_KEY` is required** — if you
+can run `claude` in your terminal, this provider works.
+
+> **Terms of use.** This provider drives your Claude Code / Agent SDK session
+> programmatically to compile wikis. That is not automatically appropriate for
+> every account type, plan, or environment. Before using it, review Anthropic's
+> current [Claude Code](https://www.anthropic.com/legal/consumer-terms) and
+> [Agent SDK](https://docs.anthropic.com/en/api/agent-sdk/overview) terms and
+> usage policies, and make sure your intended use complies with them.
+
+```bash
+export LLMWIKI_PROVIDER=claude-agent
+export LLMWIKI_MODEL=claude-sonnet-4-6  # optional; this is the default
+llmwiki compile
+```
+
+Notes:
+
+- Generation (`compile`) and structured extraction work off the local login with
+  no extra credentials.
+- Semantic search (`llmwiki query`) still needs embeddings, which Claude does not
+  provide. Set `VOYAGE_API_KEY` to enable them (same as the `anthropic`
+  provider); otherwise `query` falls back to lexical ranking.
+- To see what the SDK is doing, set `LLMWIKI_DEBUG=1` for a concise one-line trace
+  per SDK message (`[claude-agent] system:init`, `… assistant`, `… result:success`)
+  plus any `claude` subprocess errors. Use `LLMWIKI_DEBUG=verbose` to additionally
+  enable the SDK's full verbose logging.
+
+  ```bash
+  LLMWIKI_DEBUG=1 LLMWIKI_PROVIDER=claude-agent llmwiki compile
+  ```
+
 ### OpenAI-Compatible Local Servers
 
 Use the OpenAI provider for local OpenAI-compatible servers such as
@@ -180,28 +231,46 @@ A truncation warning prints to stderr when the cap fires so you know which conce
 <br>
 
 
-## Why not just RAG?
+## Why compile, not just retrieve?
+
+llmwiki uses embeddings — chunk-level, incremental, with BM25 reranking. But the embedding layer sits **below** the compiled wiki, not in front of it.
 
-RAG retrieves chunks at query time. Every question re-discovers the same relationships from scratch. Nothing accumulates.
+**RAG retrieves chunks at query time.** Every question re-discovers the same relationships from scratch. The wiki structure, citation graph, and merged-concept disambiguation never accumulate; they get re-invented per query.
 
-llmwiki **compiles** your sources into a wiki. Concepts get their own pages. Pages link to each other. When you ask a question with `--save`, the answer becomes a new page, and future queries use it as context. Your explorations compound.
+**llmwiki compiles your sources into a wiki first.** Concepts get their own typed pages. Concepts shared across multiple sources are merged into one page instead of competing as duplicate chunks. Pages link to each other via `[[wikilinks]]`. When you ask a question with `--save`, the answer becomes a new page, and future queries use it as context.
 
-This is complementary to RAG, not a replacement. RAG is great for ad-hoc retrieval over large corpora. llmwiki gives you a persistent, structured artifact to retrieve from.
+Then semantic retrieval, BM25 reranking, and graph expansion run over the compiled artifact — narrowing hundreds of pages to a tight, citation-traceable evidence pack.
 
 ```
 RAG:     query → search chunks → answer → forget
-llmwiki: sources → compile → wiki → query → save → richer wiki → better answers
+llmwiki: sources → compile → wiki → embed → query → save → richer wiki → better answers
 ```
 
+llmwiki is complementary to traditional RAG: use RAG for ad-hoc retrieval over noisy or fast-changing corpora; use llmwiki when you want a persistent, structured, citation-traceable artifact that compounds.
+
 ## How it works
 
 ```
-sources/  →  SHA-256 hash check  →  LLM concept extraction  →  wiki page generation  →  [[wikilink]] resolution  →  index.md
+sources/  →  hash check  →  LLM concept extraction  →  page generation  →  [[wikilink]] resolve
+              │                                                             ↓
+              │                                        chunk embeddings  ←  wiki/  →  index.md
+              │                                               ↓
+              │                             semantic search + BM25 rerank + graph expansion
+              │                                               ↓
+              │                                    llmwiki query / context / MCP
+              ↓
+   stale / orphaned pages  →  llmwiki refresh --stale  →  recompile changed owners, clean up orphans
 ```
 
-**Two-phase pipeline.** Phase 1 extracts all concepts from all sources. Phase 2 generates pages. This eliminates order-dependence, catches failures before writing anything, and merges concepts shared across multiple sources into single pages.
+**Two-phase compile.** Phase 1 extracts all concepts from every source; Phase 2 generates pages. Splitting the phases eliminates order-dependence, catches extraction failures before anything is written, merges concepts shared across multiple sources into a single page, and marks pages whose sources were all deleted as `orphaned` rather than silently dropping them.
 
-**Incremental.** Only changed sources go through the LLM. Everything else is skipped via hash-based change detection.
+**Incremental everywhere.** Hash-based change detection on sources, content-hash-aware embedding updates, and cached citation judgements mean only changed work runs through the LLM. Recompiling after editing one source touches just the pages that source contributed to.
+
+**Source freshness and repair.** Every page records the sources — and their content hashes — that produced it. On any later command, llmwiki compares those recorded hashes against `sources/` on disk: a page whose sources changed since the last compile is `stale`, and a page whose sources were all deleted is `orphaned`. `llmwiki lint`, `llmwiki status`, the viewer, the JSON export, and the MCP tools surface this without recompiling anything. `llmwiki refresh --stale` then repairs it — recompiling only the changed sources that own stale pages and cleaning up orphaned ones, while deliberately leaving unrelated new sources for a full `llmwiki compile`. `--dry-run` previews the plan with no LLM calls.
+
+**Hybrid retrieval.** `.llmwiki/embeddings.json` v2 carries page- and chunk-level vectors. `llmwiki query` and `llmwiki context` narrow hundreds of pages down to a chunk-level top-K via cosine similarity, then rerank with BM25 and expand along the wikilink graph for the final evidence pack.
+
+**Citation-traceable.** Paragraphs carry `^[source.md]` markers; specific claims pin to `^[source.md:42-58]` line ranges. `llmwiki lint` validates that every citation resolves to a real file and line range; `llmwiki eval` measures citation precision and (optionally) LLM-judged claim support.
 
 **Compounding queries.** `llmwiki query --save` writes the answer as a wiki page and immediately rebuilds the index. Saved answers show up in future queries as context.
 
@@ -252,19 +321,24 @@ Pages include source attribution in frontmatter. Paragraphs are annotated with `
 | `llmwiki compile` | Incremental compile: extract concepts, generate wiki pages |
 | `llmwiki compile --review` | Write candidate pages to `.llmwiki/candidates/` instead of `wiki/` so you can review before they land |
 | `llmwiki compile --lang <code>` | Generate wiki content in the given language (e.g. `Chinese`, `ja`, `zh-CN`); also works on `query` |
+| `llmwiki refresh --stale [--dry-run]` | Repair stale/orphaned pages: recompile the sources that own stale pages and clean up deleted owners, skipping unrelated new sources. `--dry-run` previews the plan with no LLM calls or writes |
 | `llmwiki review list` | List pending candidate pages |
 | `llmwiki review show <id>` | Print a candidate's title, summary, and body |
 | `llmwiki review approve <id>` | Promote a candidate into `wiki/` and refresh index/MOC/embeddings |
 | `llmwiki review reject <id>` | Archive a candidate without touching `wiki/` |
+| `llmwiki rules extract` | Extract machine-actionable rule candidates from changed sources into `.llmwiki/rule-candidates/` |
+| `llmwiki rules list` | List pending rule candidates |
+| `llmwiki rules approve <id>` / `reject <id>` | Approve or reject a rule candidate |
+| `llmwiki rules export` | Emit approved rule candidates as a JSON array for a downstream rule importer |
 | `llmwiki schema init` | Write a starter `.llmwiki/schema.json` file |
 | `llmwiki schema show` | Print the resolved schema for the current project |
 | `llmwiki query "question"` | Ask questions against your compiled wiki |
 | `llmwiki query "question" --save` | Answer and save the result as a wiki page |
-| `llmwiki export [--target <name>]` | Export the wiki to portable formats — `llms.txt`, `llms-full.txt`, JSON, JSON-LD, GraphML, Marp slides |
+| `llmwiki export [--target <name>] [--project-id <id>]` | Export the wiki to portable formats — `llms.txt`, `llms-full.txt`, JSON, JSON-LD, GraphML, Marp slides. `--project-id` pins a stable identifier inside the JSON envelope so downstream importers (e.g. [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki)) can derive deterministic external IDs |
 | `llmwiki view [--open]` | Start a read-only local web viewer for browsing, searching, and inspecting the compiled wiki |
 | `llmwiki next [--json]` | Show the recommended next action for this project (read-only); `--json` emits a stable envelope for agents |
 | `llmwiki context "<prompt>" [--json]` | Build an agent-ready evidence pack (primary pages, citations, neighbors, suggested actions) — same v1 envelope as MCP `get_context_pack` |
-| `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, low confidence, contradictions, etc.) |
+| `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, low confidence, contradictions, stale pages whose sources changed, etc.) |
 | `llmwiki eval [--suite fast\|full]` | Measure wiki quality: health score (0–100), citation coverage, corpus stats. `--suite full` adds LLM-as-judge citation support scoring |
 | `llmwiki eval cache show` | Print score distribution and top-cited pages from the citation judgement cache |
 | `llmwiki eval cache clear` | Remove the citation judgement cache |
@@ -279,6 +353,7 @@ Pages include source attribution in frontmatter. Paragraphs are annotated with `
 ## Output
 
 ```
+log.md              append-only activity journal (ingests, compiles, queries)
 wiki/
   concepts/         one .md file per concept, with YAML frontmatter
   queries/          saved query answers, included in index and retrieval
@@ -289,7 +364,31 @@ wiki/
   candidates/archive/  rejected candidates kept for audit
 ```
 
-Obsidian-compatible. `[[wikilinks]]` resolve to concept titles.
+Obsidian-compatible. `[[wikilinks]]` resolve to concept titles — or to any page that declares the term in its `aliases` frontmatter, so links survive renames and synonyms.
+
+`log.md` records what happened and when. Each entry is a heading with a fixed
+prefix — `## [YYYY-MM-DDThh:mm:ssZ] operation | description` (an ISO 8601 UTC
+timestamp) — followed by a short bullet body carrying page wikilinks and counts:
+
+```markdown
+## [2026-06-05T09:14:02Z] ingest | Attention Is All You Need
+- Source: https://arxiv.org/abs/1706.03762
+- Saved: sources/attention-is-all-you-need.md
+- Chars: 38,214
+
+## [2026-06-05T09:15:30Z] compile | 1 source(s) → 6 page(s)
+- Sources: attention-is-all-you-need.md
+- Created: [[self-attention]], [[multi-head-attention]], [[transformer]]
+- Updated: [[positional-encoding]]
+
+## [2026-06-05T09:16:11Z] query | What is multi-head attention?
+- Pages: [[multi-head-attention]], [[self-attention]]
+```
+
+Only headings start with `## [`, so the gist's recipe still works even with the
+bodies: `grep "^## \[" log.md | tail -5` shows the five most recent operations.
+Where `index.md` organizes content for discovery, `log.md` tracks temporal
+progression.
 
 ## Local web viewer
 
@@ -403,6 +502,7 @@ llmwiki eval cache clear       # wipe the citation judgement cache
 - **Health score (0–100)** aggregates all lint rules. Errors (broken citations, broken wikilinks, duplicate concepts) cost more than warnings.
 - **Citation coverage** — fraction of prose paragraphs that carry a `^[...]` marker, plus citation precision (fraction of citations pointing to existing source files).
 - **Citation support (full suite)** — samples up to N `(claim, source span)` pairs, asks a judge model to score each 0–2 (unsupported → fully supported), and caches results so subsequent runs only re-judge new pairs.
+- **Source utilization & citation depth** — the fraction of a page's valid sources that are actually cited (`source_utilization_rate`), and the share of citations pinned to specific line ranges rather than whole files (`claim_level_citation_rate`). Source warnings flag sources excluded from compilation (e.g. out-of-tree symlinks), gateable via `source_warnings_max`.
 - **Corpus stats** — page count, source count, total wiki characters, embedding counts, appended to `history.jsonl` for trend tracking.
 - **Regression deltas** — current report is diffed against the previous entry in history.
 
@@ -413,6 +513,9 @@ health_score: 85
 citation_coverage_percent: 70
 citation_precision_percent: 90
 citation_support_mean: 1.4   # only checked when --suite full
+source_utilization_rate: 0.9 # min fraction of valid sources cited by a page
+source_warnings_max: 0       # max excluded sources (out-of-tree symlinks, etc.)
+claim_level_citation_rate: 0.5 # min fraction of citations with line ranges
 ```
 
 Threshold violations are listed in the report. Exit code is non-zero when any threshold is breached, suitable for CI gating.
@@ -504,8 +607,9 @@ Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for
 | `search_pages` | Return full content of pages relevant to a question. |
 | `read_page` | Read a single page by slug (concepts/ then queries/). |
 | `lint_wiki` | Run quality checks; returns structured diagnostics. |
-| `wiki_status` | Page count, source count, orphans, pending changes (read-only). |
-| `get_context_pack` | Build an agent-ready evidence pack (primary pages, semantic chunks, graph neighbors, citations, warnings, suggested actions) — same v1 JSON envelope as `llmwiki context --json`. `get_context_pack` **packages evidence**; `query_wiki` **generates answers**. |
+| `wiki_status` | Page/source counts, stale and orphaned pages, a `stateStatus` field, and pending changes (read-only). |
+| `get_context_pack` | Build an agent-ready evidence pack (primary pages, semantic chunks, graph neighbors, citations, per-page freshness, warnings, suggested actions) — same v1 JSON envelope as `llmwiki context --json`. `get_context_pack` **packages evidence**; `query_wiki` **generates answers**. |
+| `run_eval` | Score wiki quality (the fast suite needs no API key; the full suite LLM-judges a sample of citations); read-only. |
 
 ### Resources
 
@@ -516,6 +620,56 @@ Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for
 | `llmwiki://query/{slug}` | A single saved query page. |
 | `llmwiki://sources` | List of ingested source files with metadata. |
 | `llmwiki://state` | Compilation state (per-source hashes, last compile times). |
+| `llmwiki://eval/report` | The most recent eval report. |
+| `llmwiki://eval/history` | Trend of past eval runs. |
+
+</details>
+
+
+<br>
+
+---
+
+<br>
+
+
+<details>
+<summary><span style="font-size: 1.4em;"><strong>SDK — programmatic API — click to expand</strong></span></summary>
+
+
+## SDK — `createWiki()`
+
+Drive llmwiki in-process instead of shelling out to the CLI. `createWiki({ root })` returns a `Wiki` facade bound to a project directory. Every method runs silently (no console output) and is concurrency-safe — quiet mode is scoped per async call, not via a global flag. `createWiki` is exported from the package entry, so `import { createWiki } from "llm-wiki-compiler"` works for any installed version.
+
+```ts
+import { createWiki } from "llm-wiki-compiler";
+
+const wiki = createWiki({ root: "./my-wiki" });
+
+await wiki.ingestText({ title: "Notes", text: "Raw text to compile…" });
+await wiki.compile();                        // needs LLM credentials
+const { answer } = await wiki.query("What did I note about X?");
+const status = await wiki.status();          // no credentials needed
+```
+
+### Methods
+
+| Method | What it does | LLM creds |
+|--------|--------------|:---------:|
+| `ingest({ source })` | Fetch a URL or read a local file into `sources/`. **Trusted input only** — a server-side fetch + local-file-read primitive (SSRF risk); use `ingestText` for untrusted content. | No |
+| `ingestText({ title, text })` | Ingest raw text — the safe path for untrusted content (no fetch, no file read). | No |
+| `compile(options?)` | Compile pending sources into wiki pages. `options.review` queues candidates instead of writing. **Sends source content to the provider.** | Yes |
+| `search(question)` | Retrieve and hydrate the most relevant page records. | Yes |
+| `query(question, options?)` | Grounded answer; `options.save` persists it as a page, `options.debug` returns retrieval detail. | Yes |
+| `getPage(ref)` / `listPages(options?)` | Read one page / list pages with filters and cursor pagination. | No |
+| `listSources(options?)` / `getSource(id)` / `deleteSource(id)` | List, read, or delete ingested sources. `id` is the `IngestResult.filename` (e.g. `"note.md"`); `deleteSource` reconciles the compiled page on the next `compile()`. | No |
+| `status()` | Read-only status snapshot — counts, freshness, pending changes. | No |
+| `lint()` | Run all lint rules; severity-counted summary. | No |
+| `getContextPack({ prompt, budget?, depth?, topPages?, topChunks? })` | Build a v1 context pack — same envelope as MCP `get_context_pack`. Semantic retrieval when embeddings exist, lexical fallback otherwise. | No |
+| `exportJson(options?)` | Structured JSON export document (same shape as `llmwiki export --target json`). | No |
+| `runEval({ mode, record? })` | Eval harness. `mode: "fast"` is credential-free; `"full"` LLM-judges a sample of citations. | full only |
+
+**Notes.** Methods that need a provider throw `ProviderUnavailableError` when no credentials are configured; the rest run credential-free. Output is suppressed and there is no progress callback in v1 (`compile`/`runEval` on a large corpus can run for minutes with no feedback). `status()`, `lint()`, and `exportJson()` each hash the full source corpus per call (no cross-call cache) — avoid calling them in a hot loop. All result types (`CompileResult`, `QueryResult`, `WikiStatus`, `ContextPack`, …) are exported from the package for typed consumption.
 
 </details>
 
@@ -527,31 +681,60 @@ Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for
 <br>
 
 
-## Limitations
+## Companion: Atomic Memory
+
+llmwiki and [Atomic Memory](https://github.com/atomicstrata/atomicmemory) are complementary layers of open context infrastructure, both maintained by [Atomic Strata](https://github.com/atomicstrata):
+
+- **llmwiki** gives you a persistent **knowledge base** — durable markdown compiled from your sources, inspectable on disk.
+- **Atomic Memory** gives your agents persistent **working memory** — runtime context that's searchable, correctable, scoped, and inspectable over time.
+
+Use them independently or together. Each remains valuable on its own — llmwiki as a notebook, RAG index, CI-checked knowledge base, or domain pack source; Atomic Memory as a runtime memory layer for any agent or app.
+
+The [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki) bridge ingests `llmwiki export --target json --project-id <id>` envelopes as one verbatim Atomic Memory record per wiki page, preserving all advisory metadata (kind, citations, confidence, provenance state, contradictions, aliases, freshness) under `memory.metadata.llmwiki.*`. See the [bridge cookbook](https://github.com/atomicstrata/atomicmemory/blob/main/packages/llmwiki/docs/cookbook.md) for the full compile → export → import → package workflow.
+
+## Scale and what works
 
-Early software. Best for small, high-signal corpora (a few dozen sources). Query routing is index-based.
+Still early software, but the scale story has matured well past the "few dozen sources" era.
 
-**Honest about truncation.** Sources that exceed the character limit are truncated on ingest with `truncated: true` and the original character count recorded in frontmatter, so downstream consumers know they're working with partial content.
+- **Semantic chunk retrieval** (`.llmwiki/embeddings.json` v2) narrows hundreds of pages down to a small top-K before LLM selection, with BM25 reranking and graph-neighborhood expansion layered on top.
+- **Incremental everything.** Hash-based source-change detection, content-hash-aware embedding updates, cached citation judgements. Re-running on an unchanged corpus is a few seconds.
+- **Lexical fallback.** Index-based routing kicks in automatically when no embedding store is present or the active provider has no embedding credentials, surfacing a stable warning code rather than hard-failing.
+
+**Honest about truncation.** Sources that exceed the character limit are truncated on ingest with `truncated: true` and the original character count recorded in frontmatter, so downstream consumers know they're working with partial content. A per-concept prompt budget prevents popular shared concepts from crashing compile.
+
+**Where it's still early.** No source-freshness watchdog yet (re-ingest detects content changes, but doesn't proactively re-check URLs). No team / multi-writer conflict resolution. The viewer is read-only by design — write operations go through the CLI or MCP.
 
 ## Karpathy's LLM Wiki pattern vs this compiler
 
-Karpathy describes an abstract pattern for turning raw data into compiled knowledge. Here's how llmwiki maps to it:
+Karpathy described an abstract pattern for turning raw data into compiled knowledge. Here's how llmwiki maps to it today:
 
 | Karpathy's concept | llmwiki | Status |
 |---|---|---|
-| Data ingest | `llmwiki ingest` | Implemented |
-| Compile wiki | `llmwiki compile` | Implemented |
-| Q&A | `llmwiki query` | Implemented |
+| Data ingest | `llmwiki ingest`, `ingest-session` (Claude/Codex/Cursor) | Implemented |
+| Compile wiki | `llmwiki compile` (two-phase, incremental) | Implemented |
+| Q&A | `llmwiki query` (semantic + BM25 + graph expansion) | Implemented |
 | Output filing (save answers back) | `llmwiki query --save` | Implemented |
 | Auto-recompile | `llmwiki watch` | Implemented |
-| Linting / health-check pass | `llmwiki lint` | Implemented |
-| Agent integration | `llmwiki serve` (MCP server) | Implemented |
-| Image support | `llmwiki ingest <image>` | Implemented |
+| Linting / health-check pass | `llmwiki lint` + `llmwiki eval` (CI-gateable) | Implemented |
+| Agent integration | `llmwiki serve` MCP server with `get_context_pack` | Implemented |
+| Multimodal ingest | Images, PDFs, transcripts via `llmwiki ingest` | Implemented |
 | Marp slides | `llmwiki export --target marp` | Implemented |
+| Bridge to runtime memory | `llmwiki export --target json --project-id` → [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki) | Implemented |
 | Fine-tuning | — | Not yet implemented |
 
 ## Roadmap
 
+Shipped in 0.9.0:
+
+- ✅ Source freshness — `llmwiki lint` flags pages whose sources changed (`stale`) or were all deleted (`orphaned`) since compile, surfaced across MCP (`wiki_status`, `get_context_pack`), context packs, the viewer (badges, a per-axis filter, health counts, a corrupt-state banner), the JSON export, and `llmwiki next`
+- ✅ `llmwiki refresh --stale` — repairs stale/orphaned pages with a targeted recompile of their changed owning sources (and deleted-owner cleanup), skipping unrelated new sources; `--dry-run` previews with no LLM calls or writes
+- ✅ JSON export bridge contract — `llmwiki export --target json --project-id <id>` adds per-page `path`, `kind`, advisory confidence/provenance, flattened citations, aliases, and freshness so downstream importers (e.g. [`@atomicmemory/llmwiki`](https://github.com/atomicstrata/atomicmemory/tree/main/packages/llmwiki)) can ingest pages as durable memory records
+- ✅ In-process SDK — `createWiki()` exposes the compiler in-process, with source-backed write APIs for programmatic callers
+- ✅ Eval over MCP + richer metrics — `run_eval` tool and read-only `llmwiki://eval/report`/`llmwiki://eval/history` resources, plus source-utilization and citation-depth metrics with a `source_warnings_max` CI gate
+- ✅ Rule-candidate extraction — extract reusable rule candidates from sources with review/approve and a JSON export pipeline
+- ✅ Claude Agent provider — authenticates through a local Claude Code login (bundled plan tokens, no separate API key)
+- ✅ Alias-aware wikilinks — the viewer resolves a `[[term]]` link to any page that declares `term` in its `aliases` frontmatter, not just an exact slug match
+
 Shipped in 0.8.0:
 
 - ✅ Guided project flow — `llmwiki next` recommends the next useful command, and `llmwiki quickstart <source>` ingests, compiles, and opens the viewer in one step
@@ -619,6 +802,10 @@ Explicitly not planned (good ideas, just not for this repo): full static-site ge
 
 Node.js >= 24, plus provider credentials (for Anthropic: `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`).
 
+## About
+
+llmwiki is maintained by [Atomic Strata](https://github.com/atomicstrata), the team behind [Atomic Memory](https://github.com/atomicstrata/atomicmemory). Atomic Strata builds open context infrastructure: durable compiled knowledge with llmwiki, runtime memory with Atomic Memory.
+
 ## License
 
 MIT