llm-wiki-compiler 0.2.0 → 0.4.0

package/README.md

@@ -66,6 +66,52 @@ Example with zero exports (Claude Code already configured):
 llmwiki compile
 ```
 
+### OpenAI-Compatible Local Servers
+
+Use the OpenAI provider for local OpenAI-compatible servers such as
+`llama-server`. `OPENAI_BASE_URL` is used for chat/tool calls, and
+`OPENAI_EMBEDDINGS_BASE_URL` is optional. Set it only when embeddings are
+served from a different endpoint; when unset, embeddings use the same client
+and base URL as chat. Include `/v1` in custom URLs.
+
+Split endpoint example:
+
+```bash
+export LLMWIKI_PROVIDER=openai
+export LLMWIKI_MODEL=qwen3.6-35b
+export LLMWIKI_EMBEDDING_MODEL=text-embedding-model
+export OPENAI_API_KEY=sk-local
+export OPENAI_BASE_URL=http://host_url:port/v1
+export OPENAI_EMBEDDINGS_BASE_URL=http://host_url:port/v1
+```
+
+`OPENAI_API_KEY` is still required by the CLI and OpenAI SDK. For local
+servers that do not check authentication, any dummy value is sufficient.
+
+### Ollama
+
+Ollama uses its OpenAI-compatible endpoint. Set `OLLAMA_HOST` for chat and
+optionally set `OLLAMA_EMBEDDINGS_HOST` only when embeddings are served from a
+different endpoint. When unset, embeddings use `OLLAMA_HOST`. Include `/v1` in
+custom URLs.
+
+```bash
+export LLMWIKI_PROVIDER=ollama
+export LLMWIKI_MODEL=llama3.1
+export LLMWIKI_EMBEDDING_MODEL=nomic-embed-text
+export OLLAMA_HOST=http://ollama_host:11434/v1
+export OLLAMA_EMBEDDINGS_HOST=http://ollama_host:11435/v1
+```
+
+### Request timeouts
+
+The OpenAI SDK defaults to a 10-minute per-request timeout, which can cut off long compile-time completions on slower local models. Override per provider:
+
+- `LLMWIKI_REQUEST_TIMEOUT_MS` — provider-agnostic timeout in milliseconds. Applies to both the `openai` and `ollama` backends.
+- `OLLAMA_TIMEOUT_MS` — Ollama-specific override. Wins over `LLMWIKI_REQUEST_TIMEOUT_MS` when both are set.
+
+Defaults: 10 minutes for `openai`, 30 minutes for `ollama` (local models commonly need more).
+
 ## Why not just RAG?
 
 RAG retrieves chunks at query time. Every question re-discovers the same relationships from scratch. Nothing accumulates.
@@ -99,6 +145,7 @@ A raw source like a Wikipedia article on knowledge compilation becomes a structu
 ---
 title: Knowledge Compilation
 summary: Techniques for converting knowledge representations into forms that support efficient reasoning.
+kind: concept
 sources:
   - knowledge-compilation.md
 createdAt: "2026-04-05T12:00:00Z"
@@ -113,7 +160,7 @@ a knowledge base into a target language that supports efficient queries.
 Related concepts: [[Propositional Logic]], [[Model Counting]]
 ```
 
-Pages include source attribution in frontmatter. Paragraphs are annotated with `^[filename.md]` markers pointing back to the source file that contributed the content.
+Pages include source attribution in frontmatter. Paragraphs are annotated with `^[filename.md]` markers pointing back to the source file that contributed the content; specific claims can use line ranges like `^[filename.md:42-58]` or `^[filename.md#L42-L58]`.
 
 ## Commands
 
@@ -121,9 +168,16 @@ Pages include source attribution in frontmatter. Paragraphs are annotated with `
 |---------|-------------|
 | `llmwiki ingest <url\|file>` | Fetch a URL or copy a local file into `sources/` |
 | `llmwiki compile` | Incremental compile: extract concepts, generate wiki pages |
+| `llmwiki compile --review` | Write candidate pages to `.llmwiki/candidates/` instead of `wiki/` so you can review before they land |
+| `llmwiki review list` | List pending candidate pages |
+| `llmwiki review show <id>` | Print a candidate's title, summary, and body |
+| `llmwiki review approve <id>` | Promote a candidate into `wiki/` and refresh index/MOC/embeddings |
+| `llmwiki review reject <id>` | Archive a candidate without touching `wiki/` |
+| `llmwiki schema init` | Write a starter `.llmwiki/schema.json` file |
+| `llmwiki schema show` | Print the resolved schema for the current project |
 | `llmwiki query "question"` | Ask questions against your compiled wiki |
 | `llmwiki query "question" --save` | Answer and save the result as a wiki page |
-| `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, etc.) |
+| `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, low confidence, contradictions, etc.) |
 | `llmwiki watch` | Auto-recompile when `sources/` changes |
 | `llmwiki serve [--root <dir>]` | Start an MCP server exposing wiki tools to AI agents |
 
@@ -131,13 +185,97 @@ Pages include source attribution in frontmatter. Paragraphs are annotated with `
 
 ```
 wiki/
-  concepts/     one .md file per concept, with YAML frontmatter
-  queries/      saved query answers, included in index and retrieval
-  index.md      auto-generated table of contents
+  concepts/         one .md file per concept, with YAML frontmatter
+  queries/          saved query answers, included in index and retrieval
+  index.md          auto-generated table of contents
+.llmwiki/
+  schema.json       optional page-kind and cross-link policy
+  candidates/       pending review candidates from `compile --review`
+  candidates/archive/  rejected candidates kept for audit
 ```
 
 Obsidian-compatible. `[[wikilinks]]` resolve to concept titles.
 
+## Review queue
+
+By default, `compile` writes pages directly to `wiki/`. Add `--review` to write candidate JSON records to `.llmwiki/candidates/` instead, so you can inspect each generated page before it lands.
+
+```bash
+llmwiki compile --review     # produces candidates, leaves wiki/ untouched
+llmwiki review list          # see what's pending
+llmwiki review show <id>     # inspect a single candidate
+llmwiki review approve <id>  # write into wiki/ + refresh index/MOC/embeddings
+llmwiki review reject <id>   # archive to .llmwiki/candidates/archive/
+```
+
+A few things to know:
+
+- **Approve and reject acquire `.llmwiki/lock`** so they serialize cleanly against each other and against any concurrent `compile`.
+- **Source state is deferred per-source.** When one source produces multiple candidates, the source isn't marked compiled until the last candidate is approved — so unresolved siblings stay re-detectable on the next `compile --review`.
+- **Deletion bookkeeping is deferred.** `compile --review` does not orphan-mark deleted sources; the next non-review `compile` does that. The `--review` help text advertises this.
+- MCP `wiki_status` exposes `pendingCandidates` so agents can see the queue depth.
+
+## Page metadata
+
+Compiled pages can carry epistemic metadata in frontmatter so consumers know how trustworthy each page is. All fields are optional and existing pages without them continue to work.
+
+```yaml
+---
+title: Knowledge Compilation
+summary: Techniques for converting knowledge representations...
+sources:
+  - knowledge-compilation.md
+confidence: 0.82           # 0–1, LLM-reported confidence in the synthesized page
+provenanceState: merged    # extracted | merged | inferred | ambiguous
+contradictedBy:
+  - slug: probabilistic-reasoning
+inferredParagraphs: 1      # paragraphs the LLM marked as inferred (vs cited)
+---
+```
+
+When multiple sources merge into one slug, metadata is reconciled: `min` confidence, `provenanceState = 'merged'`, union of `contradictedBy` (deduped by slug), `max` `inferredParagraphs`.
+
+`llmwiki lint` adds three rules that surface this metadata:
+
+- `low-confidence` — flags pages with `confidence` below a threshold
+- `contradicted-page` — flags pages with non-empty `contradictedBy`
+- `excess-inferred-paragraphs` — flags pages with too many inferred paragraphs without citations
+
+## Claim-level provenance
+
+Paragraph citations continue to use the original source-marker form:
+
+```markdown
+This paragraph is grounded in the source. ^[source.md]
+```
+
+For claims that need tighter verification, pages can pin a statement to a line range in the ingested source:
+
+```markdown
+The system uses a two-phase compile pipeline. ^[architecture-notes.md:42-58]
+The same range can also use GitHub-style anchors. ^[architecture-notes.md#L42-L58]
+```
+
+`llmwiki lint` validates both forms. It reports missing source files, malformed claim citations, impossible ranges like line `0` or `8-3`, and ranges that extend past the end of the source file.
+
+## Schema layer
+
+Projects can optionally define `.llmwiki/schema.json` to shape the wiki beyond flat concept pages. Existing projects do not need a schema file; missing or invalid `kind` values fall back to `concept`.
+
+```bash
+llmwiki schema init
+llmwiki schema show
+```
+
+The schema supports four page kinds:
+
+- `concept` — standalone idea or pattern
+- `entity` — specific person, product, organization, or named artifact
+- `comparison` — side-by-side analysis across concepts or entities
+- `overview` — map page that connects several concepts in a domain
+
+Schema rules can set per-kind `minWikilinks` and optional `seedPages`. Compile can materialize seed pages such as overviews, lint enforces page-kind-specific cross-link minimums, and review candidates surface schema violations before approval.
+
 ## Demo
 
 Try it on any article or document:
@@ -232,17 +370,33 @@ Karpathy describes an abstract pattern for turning raw data into compiled knowle
 
 ## Roadmap
 
+Shipped in 0.4.0:
+
+- ✅ Claim-level provenance with source ranges
+- ✅ First-class schema layer with typed page kinds (`concept`, `entity`, `comparison`, `overview`)
+
+Shipped in 0.3.0:
+
+- ✅ Candidate review queue (approve compile output before pages are written)
+- ✅ Confidence and contradiction metadata on compiled pages
+
+Shipped in 0.2.0:
+
 - ✅ Better provenance (paragraph-level source attribution)
 - ✅ Linting pass for wiki quality checks
 - ✅ Multi-provider support (OpenAI, Ollama, MiniMax)
 - ✅ Larger-corpus query strategy (semantic search, embeddings)
 - ✅ Deeper Obsidian integration (tags, aliases, Map of Content)
 - ✅ MCP server for agent integration
-- Image support
-- Marp slides
-- Fine-tuning
 
-If you want to contribute, these are the highest-leverage areas right now. Issues and PRs are welcome.
+Next up:
+
+- Multimodal ingest (images, PDFs, transcripts)
+- Chunked retrieval with reranking
+- Export bundle (`llms.txt`, JSON, JSON-LD, GraphML, Marp)
+- Session-history adapters (Claude, Codex, Cursor exports)
+
+If you like ambitious problems: **multimodal ingest**, **chunked retrieval with reranking**, and **export bundles** are the meatiest. Open an issue to claim one or kick off a design discussion.
 
 ## Requirements