llm-wiki-compiler 0.1.1 → 0.3.0

package/README.md

@@ -17,12 +17,92 @@ Inspired by Karpathy's [LLM Wiki](https://gist.github.com/karpathy/442a6bf555914
 ```bash
 npm install -g llm-wiki-compiler
 export ANTHROPIC_API_KEY=sk-...
+# Or use ANTHROPIC_AUTH_TOKEN if your Anthropic-compatible gateway expects it.
+# Or use a different provider:
+# export LLMWIKI_PROVIDER=openai
+# export OPENAI_API_KEY=sk-...
 
 llmwiki ingest https://some-article.com
 llmwiki compile
 llmwiki query "what is X?"
 ```
 
+## Configuration
+
+llmwiki configures providers via environment variables. The default provider is Anthropic.
+
+Configuration precedence for Anthropic values:
+
+1. Shell env / local `.env`
+2. Claude Code settings fallback (`~/.claude/settings.json` → `env` block)
+3. Built-in provider defaults (where applicable)
+
+- `LLMWIKI_PROVIDER`: The provider to use (e.g., anthropic, openai).
+- `LLMWIKI_MODEL`: The model name to override the provider default.
+
+### Anthropic (Default)
+
+- `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`: Required. Either one can satisfy Anthropic authentication.
+- `ANTHROPIC_BASE_URL`: Optional. Custom endpoint for proxies. Valid HTTP(S) URLs are accepted, including Claude-style path endpoints such as `https://api.kimi.com/coding/`.
+
+Example using an Anthropic or cc-switch custom proxy:
+
+```bash
+export LLMWIKI_PROVIDER=anthropic
+export ANTHROPIC_API_KEY=sk-...
+export ANTHROPIC_BASE_URL=https://proxy.example.com
+```
+
+If those values are not set in shell env or `.env`, llmwiki will try Anthropic-compatible values from `~/.claude/settings.json` (`env` block) for:
+
+- `ANTHROPIC_API_KEY`
+- `ANTHROPIC_AUTH_TOKEN`
+- `ANTHROPIC_BASE_URL`
+- `ANTHROPIC_MODEL`
+
+Example with zero exports (Claude Code already configured):
+
+```bash
+llmwiki compile
+```
+
+### OpenAI-Compatible Local Servers
+
+Use the OpenAI provider for local OpenAI-compatible servers such as
+`llama-server`. `OPENAI_BASE_URL` is used for chat/tool calls, and
+`OPENAI_EMBEDDINGS_BASE_URL` is optional. Set it only when embeddings are
+served from a different endpoint; when unset, embeddings use the same client
+and base URL as chat. Include `/v1` in custom URLs.
+
+Split endpoint example:
+
+```bash
+export LLMWIKI_PROVIDER=openai
+export LLMWIKI_MODEL=qwen3.6-35b
+export LLMWIKI_EMBEDDING_MODEL=text-embedding-model
+export OPENAI_API_KEY=sk-local
+export OPENAI_BASE_URL=http://host_url:port/v1
+export OPENAI_EMBEDDINGS_BASE_URL=http://host_url:port/v1
+```
+
+`OPENAI_API_KEY` is still required by the CLI and OpenAI SDK. For local
+servers that do not check authentication, any dummy value is sufficient.
+
+### Ollama
+
+Ollama uses its OpenAI-compatible endpoint. Set `OLLAMA_HOST` for chat and
+optionally set `OLLAMA_EMBEDDINGS_HOST` only when embeddings are served from a
+different endpoint. When unset, embeddings use `OLLAMA_HOST`. Include `/v1` in
+custom URLs.
+
+```bash
+export LLMWIKI_PROVIDER=ollama
+export LLMWIKI_MODEL=llama3.1
+export LLMWIKI_EMBEDDING_MODEL=nomic-embed-text
+export OLLAMA_HOST=http://ollama_host:11434/v1
+export OLLAMA_EMBEDDINGS_HOST=http://ollama_host:11435/v1
+```
+
 ## Why not just RAG?
 
 RAG retrieves chunks at query time. Every question re-discovers the same relationships from scratch. Nothing accumulates.
@@ -70,7 +150,7 @@ a knowledge base into a target language that supports efficient queries.
 Related concepts: [[Propositional Logic]], [[Model Counting]]
 ```
 
-Pages include source attribution in frontmatter. Provenance is page-level today, not claim-level.
+Pages include source attribution in frontmatter. Paragraphs are annotated with `^[filename.md]` markers pointing back to the source file that contributed the content.
 
 ## Commands
 
@@ -78,21 +158,76 @@ Pages include source attribution in frontmatter. Provenance is page-level today,
 |---------|-------------|
 | `llmwiki ingest <url\|file>` | Fetch a URL or copy a local file into `sources/` |
 | `llmwiki compile` | Incremental compile: extract concepts, generate wiki pages |
+| `llmwiki compile --review` | Write candidate pages to `.llmwiki/candidates/` instead of `wiki/` so you can review before they land |
+| `llmwiki review list` | List pending candidate pages |
+| `llmwiki review show <id>` | Print a candidate's title, summary, and body |
+| `llmwiki review approve <id>` | Promote a candidate into `wiki/` and refresh index/MOC/embeddings |
+| `llmwiki review reject <id>` | Archive a candidate without touching `wiki/` |
 | `llmwiki query "question"` | Ask questions against your compiled wiki |
 | `llmwiki query "question" --save` | Answer and save the result as a wiki page |
+| `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, low confidence, contradictions, etc.) |
 | `llmwiki watch` | Auto-recompile when `sources/` changes |
+| `llmwiki serve [--root <dir>]` | Start an MCP server exposing wiki tools to AI agents |
 
 ## Output
 
 ```
 wiki/
-  concepts/     one .md file per concept, with YAML frontmatter
-  queries/      saved query answers, included in index and retrieval
-  index.md      auto-generated table of contents
+  concepts/         one .md file per concept, with YAML frontmatter
+  queries/          saved query answers, included in index and retrieval
+  index.md          auto-generated table of contents
+.llmwiki/
+  candidates/       pending review candidates from `compile --review`
+  candidates/archive/  rejected candidates kept for audit
 ```
 
 Obsidian-compatible. `[[wikilinks]]` resolve to concept titles.
 
+## Review queue
+
+By default, `compile` writes pages directly to `wiki/`. Add `--review` to write candidate JSON records to `.llmwiki/candidates/` instead, so you can inspect each generated page before it lands.
+
+```bash
+llmwiki compile --review     # produces candidates, leaves wiki/ untouched
+llmwiki review list          # see what's pending
+llmwiki review show <id>     # inspect a single candidate
+llmwiki review approve <id>  # write into wiki/ + refresh index/MOC/embeddings
+llmwiki review reject <id>   # archive to .llmwiki/candidates/archive/
+```
+
+A few things to know:
+
+- **Approve and reject acquire `.llmwiki/lock`** so they serialize cleanly against each other and against any concurrent `compile`.
+- **Source state is deferred per-source.** When one source produces multiple candidates, the source isn't marked compiled until the last candidate is approved — so unresolved siblings stay re-detectable on the next `compile --review`.
+- **Deletion bookkeeping is deferred.** `compile --review` does not orphan-mark deleted sources; the next non-review `compile` does that. The `--review` help text advertises this.
+- MCP `wiki_status` exposes `pendingCandidates` so agents can see the queue depth.
+
+## Page metadata
+
+Compiled pages can carry epistemic metadata in frontmatter so consumers know how trustworthy each page is. All fields are optional and existing pages without them continue to work.
+
+```yaml
+---
+title: Knowledge Compilation
+summary: Techniques for converting knowledge representations...
+sources:
+  - knowledge-compilation.md
+confidence: 0.82           # 0–1, LLM-reported confidence in the synthesized page
+provenanceState: merged    # extracted | merged | inferred | ambiguous
+contradictedBy:
+  - slug: probabilistic-reasoning
+inferredParagraphs: 1      # paragraphs the LLM marked as inferred (vs cited)
+---
+```
+
+When multiple sources merge into one slug, metadata is reconciled: `min` confidence, `provenanceState = 'merged'`, union of `contradictedBy` (deduped by slug), `max` `inferredParagraphs`.
+
+`llmwiki lint` adds three rules that surface this metadata:
+
+- `low-confidence` — flags pages with `confidence` below a threshold
+- `contradicted-page` — flags pages with non-empty `contradictedBy`
+- `excess-inferred-paragraphs` — flags pages with too many inferred paragraphs without citations
+
 ## Demo
 
 Try it on any article or document:
@@ -106,9 +241,65 @@ llmwiki query "What terms did Andrej coin?"
 
 See `examples/basic/` in the repo for pre-generated output you can browse without an API key.
 
+## MCP Server
+
+llmwiki ships an MCP (Model Context Protocol) server so AI agents (Claude Desktop, Cursor, Claude Code, etc.) can drive the full pipeline directly: ingest sources, compile, query, search, lint, and read pages — without scraping CLI output.
+
+Where [llm-wiki-kit](https://github.com/iamsashank09/llm-wiki-kit) gives agents raw CRUD against wiki pages, llmwiki exposes the **automated pipelines**: agents get intelligent compilation, incremental change detection, and semantic query routing built in.
+
+### Setup
+
+Start the server (stdio transport, no API key required at startup):
+
+```bash
+llmwiki serve --root /path/to/your/wiki-project
+```
+
+### Claude Desktop / Cursor configuration
+
+Add to your client's MCP config (e.g. `claude_desktop_config.json`):
+
+```json
+{
+  "mcpServers": {
+    "llmwiki": {
+      "command": "npx",
+      "args": ["llm-wiki-compiler", "serve", "--root", "/path/to/wiki-project"],
+      "env": {
+        "ANTHROPIC_API_KEY": "sk-ant-..."
+      }
+    }
+  }
+}
+```
+
+Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for a configured provider on each call. Read-only tools (`read_page`, `lint_wiki`, `wiki_status`) and `ingest_source` work without any credentials.
+
+### Tools
+
+| Tool | What it does |
+|------|--------------|
+| `ingest_source` | Fetch a URL or local file into `sources/`. |
+| `compile_wiki` | Run the incremental compile pipeline; returns counts, slugs, errors. |
+| `query_wiki` | Two-step grounded answer with optional `--save`. |
+| `search_pages` | Return full content of pages relevant to a question. |
+| `read_page` | Read a single page by slug (concepts/ then queries/). |
+| `lint_wiki` | Run quality checks; returns structured diagnostics. |
+| `wiki_status` | Page count, source count, orphans, pending changes (read-only). |
+
+### Resources
+
+| URI | Returns |
+|-----|---------|
+| `llmwiki://index` | Full `wiki/index.md` content. |
+| `llmwiki://concept/{slug}` | A single concept page (frontmatter + body). |
+| `llmwiki://query/{slug}` | A single saved query page. |
+| `llmwiki://sources` | List of ingested source files with metadata. |
+| `llmwiki://state` | Compilation state (per-source hashes, last compile times). |
+
 ## Limitations
 
-Early software. Best for small, high-signal corpora (a few dozen sources). Query routing is index-based. Provenance is page-level, not claim-level. Anthropic-only for now.
+Early software. Best for small, high-signal corpora (a few dozen sources). Query routing is index-based.
 
 **Honest about truncation.** Sources that exceed the character limit are truncated on ingest with `truncated: true` and the original character count recorded in frontmatter, so downstream consumers know they're working with partial content.
 
@@ -123,24 +314,42 @@ Karpathy describes an abstract pattern for turning raw data into compiled knowle
 | Q&A | `llmwiki query` | Implemented |
 | Output filing (save answers back) | `llmwiki query --save` | Implemented |
 | Auto-recompile | `llmwiki watch` | Implemented |
-| Linting / health-check pass | — | Not yet implemented (`watch` is auto-recompile, not lint) |
+| Linting / health-check pass | `llmwiki lint` | Implemented |
+| Agent integration | `llmwiki serve` (MCP server) | Implemented |
 | Image support | — | Not yet implemented |
 | Marp slides | — | Not yet implemented |
 | Fine-tuning | — | Not yet implemented |
 
 ## Roadmap
 
-- Multi-provider support (OpenAI, local models)
-- Better provenance (claim-level source attribution)
-- Larger-corpus query strategy (semantic search, embeddings)
-- Deeper Obsidian integration
-- Linting pass for wiki quality checks
+Shipped in 0.3.0:
+
+- ✅ Candidate review queue (approve compile output before pages are written)
+- ✅ Confidence and contradiction metadata on compiled pages
+
+Shipped in 0.2.0:
+
+- ✅ Better provenance (paragraph-level source attribution)
+- ✅ Linting pass for wiki quality checks
+- ✅ Multi-provider support (OpenAI, Ollama, MiniMax)
+- ✅ Larger-corpus query strategy (semantic search, embeddings)
+- ✅ Deeper Obsidian integration (tags, aliases, Map of Content)
+- ✅ MCP server for agent integration
+
+Next up:
+
+- Claim-level provenance with source ranges
+- First-class schema layer with typed page kinds (`concept`, `entity`, `comparison`, `overview`)
+- Multimodal ingest (images, PDFs, transcripts)
+- Chunked retrieval with reranking
+- Export bundle (`llms.txt`, JSON, JSON-LD, GraphML, Marp)
+- Session-history adapters (Claude, Codex, Cursor exports)
 
-If you want to contribute, these are the highest-leverage areas right now. Issues and PRs are welcome.
+If you like ambitious problems: **schema layer + typed page kinds**, **claim-level provenance**, and **chunked retrieval with reranking** are the meatiest. Open an issue to claim one or kick off a design discussion.
 
 ## Requirements
 
-Node.js >= 18, an Anthropic API key.
+Node.js >= 18, plus provider credentials (for Anthropic: `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`).
 
 ## License