llm-wiki-compiler 0.5.1 → 0.7.0

package/README.md

@@ -25,9 +25,20 @@ export ANTHROPIC_API_KEY=sk-...
 llmwiki ingest https://some-article.com
 llmwiki compile
 llmwiki query "what is X?"
+llmwiki view --open
 ```
 
-## Configuration
+
+<br>
+
+---
+
+<br>
+
+
+<details>
+<summary><span style="font-size: 1.4em;"><strong>Configuration — click to expand</strong></span></summary>
+
 
 llmwiki configures providers via environment variables. The default provider is Anthropic.
 
@@ -103,6 +114,35 @@ export OLLAMA_HOST=http://ollama_host:11434/v1
 export OLLAMA_EMBEDDINGS_HOST=http://ollama_host:11435/v1
 ```
 
+### GitHub Copilot
+
+Uses the GitHub Copilot API (`https://api.githubcopilot.com`), an
+OpenAI-compatible endpoint available to Copilot subscribers. Requires a GitHub
+OAuth token with the `copilot` scope — **classic PATs are not supported**.
+
+First, ensure your `gh` CLI token has the required scope:
+
+```bash
+gh auth refresh --scopes copilot
+```
+
+Then run:
+
+```bash
+export LLMWIKI_PROVIDER=copilot
+export GITHUB_TOKEN=$(gh auth token)  # OAuth token required; PATs will not work
+export LLMWIKI_MODEL=gpt-4o           # optional; gpt-4o is the default
+```
+
+Available models (names use dots, not dashes): `gpt-4o`, `gpt-4o-mini`,
+`claude-sonnet-4.5`, `claude-sonnet-4.6`, `claude-opus-4.5`, `gemini-2.5-pro`,
+and others — availability depends on your Copilot plan.
+
+**Embeddings:** The GitHub Copilot API does not expose an embeddings endpoint.
+Semantic search (used by `llmwiki query` with chunked retrieval) will fall back
+to full-index selection without embeddings. For embedding-dependent workflows,
+switch to the `openai` provider and provide `OPENAI_API_KEY`.
+
 ### Request timeouts
 
 The OpenAI SDK defaults to a 10-minute per-request timeout, which can cut off long compile-time completions on slower local models. Override per provider:
@@ -112,6 +152,33 @@ The OpenAI SDK defaults to a 10-minute per-request timeout, which can cut off lo
 
 Defaults: 10 minutes for `openai`, 30 minutes for `ollama` (local models commonly need more).
 
+### Output language
+
+Generated wiki content defaults to whatever language the model produces from the source material — typically English. Override with either:
+
+- `LLMWIKI_OUTPUT_LANG` — e.g. `zh-CN`, `Chinese`, `ja`, `Japanese`. Applies to every prompt the compile and query pipelines make.
+- `--lang <code>` on `llmwiki compile` and `llmwiki query` — same effect, scoped to one invocation. Wins over the env var.
+
+Unset preserves prior behaviour byte-for-byte.
+
+### Per-concept prompt budget
+
+When many sources contribute to the same compiled concept, `compile` enforces a per-concept character cap on the combined source content sent to the LLM so popular shared concepts don't blow past the model's context window. Each contributing source gets a fair share when truncation kicks in.
+
+- `LLMWIKI_PROMPT_BUDGET_CHARS` — character ceiling for the combined per-concept prompt. Defaults to `200000` (~50k tokens), which fits modern context windows with headroom. Raise it for larger-context models, lower it for local small-context models.
+
+A truncation warning prints to stderr when the cap fires so you know which concept hit the budget.
+
+</details>
+
+
+<br>
+
+---
+
+<br>
+
+
 ## Why not just RAG?
 
 RAG retrieves chunks at query time. Every question re-discovers the same relationships from scratch. Nothing accumulates.
@@ -162,13 +229,27 @@ Related concepts: [[Propositional Logic]], [[Model Counting]]
 
 Pages include source attribution in frontmatter. Paragraphs are annotated with `^[filename.md]` markers pointing back to the source file that contributed the content; specific claims can use line ranges like `^[filename.md:42-58]` or `^[filename.md#L42-L58]`.
 
+
+<br>
+
+---
+
+<br>
+
+
+<details>
+<summary><span style="font-size: 1.4em;"><strong>CLI and wiki model — click to expand</strong></span></summary>
+
+
 ## Commands
 
 | Command | What it does |
 |---------|-------------|
 | `llmwiki ingest <url\|file>` | Fetch a URL or copy a local file into `sources/` |
+| `llmwiki ingest-session <path>` | Import a Claude/Codex/Cursor session export (single file or whole directory) into `sources/` |
 | `llmwiki compile` | Incremental compile: extract concepts, generate wiki pages |
 | `llmwiki compile --review` | Write candidate pages to `.llmwiki/candidates/` instead of `wiki/` so you can review before they land |
+| `llmwiki compile --lang <code>` | Generate wiki content in the given language (e.g. `Chinese`, `ja`, `zh-CN`); also works on `query` |
 | `llmwiki review list` | List pending candidate pages |
 | `llmwiki review show <id>` | Print a candidate's title, summary, and body |
 | `llmwiki review approve <id>` | Promote a candidate into `wiki/` and refresh index/MOC/embeddings |
@@ -177,6 +258,8 @@ Pages include source attribution in frontmatter. Paragraphs are annotated with `
 | `llmwiki schema show` | Print the resolved schema for the current project |
 | `llmwiki query "question"` | Ask questions against your compiled wiki |
 | `llmwiki query "question" --save` | Answer and save the result as a wiki page |
+| `llmwiki export [--target <name>]` | Export the wiki to portable formats — `llms.txt`, `llms-full.txt`, JSON, JSON-LD, GraphML, Marp slides |
+| `llmwiki view [--open]` | Start a read-only local web viewer for browsing, searching, and inspecting the compiled wiki |
 | `llmwiki lint` | Check wiki quality (broken links, orphans, empty pages, low confidence, contradictions, etc.) |
 | `llmwiki watch` | Auto-recompile when `sources/` changes |
 | `llmwiki serve [--root <dir>]` | Start an MCP server exposing wiki tools to AI agents |
@@ -196,6 +279,17 @@ wiki/
 
 Obsidian-compatible. `[[wikilinks]]` resolve to concept titles.
 
+## Local web viewer
+
+Run `llmwiki view` from a project root to browse the compiled wiki in a local browser without Obsidian. The viewer is read-only: it renders `wiki/`, exposes sidebar navigation, search, page metadata, health counts, and provenance/citation chips, but does not mutate sources or generated pages.
+
+```bash
+llmwiki view          # prints Viewer ready at http://127.0.0.1:<port>
+llmwiki view --open   # also opens the URL in your default browser
+```
+
+The server is private by default. It binds to `127.0.0.1` unless you explicitly provide both `--host <host>` and `--allow-lan`; wildcard hosts are rejected. Viewer responses use a strict local-asset CSP and path-confinement checks so the UI can safely render local markdown content.
+
 ## Review queue
 
 By default, `compile` writes pages directly to `wiki/`. Add `--review` to write candidate JSON records to `.llmwiki/candidates/` instead, so you can inspect each generated page before it lands.
@@ -229,17 +323,16 @@ confidence: 0.82           # 0–1, LLM-reported confidence in the synthesized p
 provenanceState: merged    # extracted | merged | inferred | ambiguous
 contradictedBy:
   - slug: probabilistic-reasoning
-inferredParagraphs: 1      # paragraphs the LLM marked as inferred (vs cited)
 ---
 ```
 
-When multiple sources merge into one slug, metadata is reconciled: `min` confidence, `provenanceState = 'merged'`, union of `contradictedBy` (deduped by slug), `max` `inferredParagraphs`.
+When multiple sources merge into one slug, metadata is reconciled: `min` confidence, `provenanceState = 'merged'`, union of `contradictedBy` (deduped by slug).
 
 `llmwiki lint` adds three rules that surface this metadata:
 
 - `low-confidence` — flags pages with `confidence` below a threshold
 - `contradicted-page` — flags pages with non-empty `contradictedBy`
-- `excess-inferred-paragraphs` — flags pages with too many inferred paragraphs without citations
+- `excess-inferred-paragraphs` — flags pages whose body has too many uncited prose paragraphs (counted directly from the rendered text — the body is the single source of truth, no frontmatter field involved)
 
 ## Claim-level provenance
 
@@ -276,6 +369,16 @@ The schema supports four page kinds:
 
 Schema rules can set per-kind `minWikilinks` and optional `seedPages`. Compile can materialize seed pages such as overviews, lint enforces page-kind-specific cross-link minimums, and review candidates surface schema violations before approval.
 
+</details>
+
+
+<br>
+
+---
+
+<br>
+
+
 ## Demo
 
 Try it on any article or document:
@@ -285,10 +388,23 @@ mkdir my-wiki && cd my-wiki
 llmwiki ingest https://en.wikipedia.org/wiki/Andrej_Karpathy
 llmwiki compile
 llmwiki query "What terms did Andrej coin?"
+llmwiki view --open
 ```
 
 See `examples/basic/` in the repo for pre-generated output you can browse without an API key.
 
+
+<br>
+
+---
+
+<br>
+
+
+<details>
+<summary><span style="font-size: 1.4em;"><strong>MCP Server — click to expand</strong></span></summary>
+
+
 ## MCP Server
 
 llmwiki ships an MCP (Model Context Protocol) server so AI agents (Claude Desktop, Cursor, Claude Code, etc.) can drive the full pipeline directly: ingest sources, compile, query, search, lint, and read pages — without scraping CLI output.
@@ -345,6 +461,16 @@ Tools that need an LLM (`compile_wiki`, `query_wiki`, `search_pages`) check for
 | `llmwiki://sources` | List of ingested source files with metadata. |
 | `llmwiki://state` | Compilation state (per-source hashes, last compile times). |
 
+</details>
+
+
+<br>
+
+---
+
+<br>
+
+
 ## Limitations
 
 Early software. Best for small, high-signal corpora (a few dozen sources). Query routing is index-based.
@@ -364,12 +490,25 @@ Karpathy describes an abstract pattern for turning raw data into compiled knowle
 | Auto-recompile | `llmwiki watch` | Implemented |
 | Linting / health-check pass | `llmwiki lint` | Implemented |
 | Agent integration | `llmwiki serve` (MCP server) | Implemented |
-| Image support | — | Not yet implemented |
-| Marp slides | — | Not yet implemented |
+| Image support | `llmwiki ingest <image>` | Implemented |
+| Marp slides | `llmwiki export --target marp` | Implemented |
 | Fine-tuning | — | Not yet implemented |
 
 ## Roadmap
 
+Shipped in 0.7.0:
+
+- ✅ Read-only local web viewer — `llmwiki view` with sidebar navigation, markdown rendering, search, metadata, health counts, and provenance/citation chips
+- ✅ GitHub Copilot provider — `LLMWIKI_PROVIDER=copilot` with `GITHUB_TOKEN=$(gh auth token)` for Copilot chat/tool calls
+- ✅ Cached lint health summary — `llmwiki lint` writes `.llmwiki/last-lint.json` so viewer health can show the latest lint counts without re-running lint
+
+Shipped in 0.6.0:
+
+- ✅ Export bundle (`llms.txt`, JSON, JSON-LD, GraphML, Marp slides)
+- ✅ Session-history adapters — `llmwiki ingest-session` for Claude, Codex, and Cursor exports
+- ✅ Configurable output language — `--lang <code>` and `LLMWIKI_OUTPUT_LANG`
+- ✅ Defensive per-concept prompt budget so popular shared concepts don't crash compile
+
 Shipped in 0.5.0:
 
 - ✅ Multimodal ingest (images, PDFs, transcripts)
@@ -397,20 +536,22 @@ Shipped in 0.2.0:
 
 Next up:
 
-- Export bundle (`llms.txt`, JSON, JSON-LD, GraphML, Marp)
-- Session-history adapters (Claude, Codex, Cursor exports)
+- **Graph/context layer** — page-neighborhood tools, graph paths, gap detection, and token-budgeted context packs for agents.
+- **Evaluation harness** — benchmark answer quality, citation accuracy, update drift, retrieval recall, and scale curves against serious retrieval baselines.
+- **Task and decision ledger** — turn session ingest into durable agent memory: goals, decisions, open questions, outcomes, and next-agent handoffs.
+- **Rollback, audit, and source lifecycle** — undo/reverse ingest, compile diff reports, stale-claim checks, freshness reports, and a durable operation log.
+- **Domain templates** — schema/prompt packs for research, codebase docs, team handbooks, decision logs, and standards/regulations.
 
-Future ideas (open to discussion):
+Later / open to discussion:
 
 - Recurring source refresh jobs — re-ingest URLs on a schedule, diff against the prior snapshot, re-compile only what changed
-- Graph export and a lightweight read-only graph browser for the concept network
-- A local read-only web UI for browsing the compiled wiki without Obsidian
-- MCP prompt resources — curated agent prompts (review the wiki, propose new sources, draft a comparison page) shipped as MCP resources
-- Maintenance log + log rotation so long-running watch sessions don't grow unbounded
+- MCP prompt resources — curated agent prompts such as "review the wiki", "propose new sources", and "draft a comparison page"
+- Codex OAuth provider — ChatGPT subscription auth as a dedicated provider, with clear token refresh and embedding-limit behavior
+- Team-chat connectors for Slack/Discord/Teams-style institutional memory
 
-If you like ambitious problems: **graph export with a browser**, **recurring source refresh**, and **MCP prompt resources** are the meatiest of the futures. Open an issue to claim one or kick off a design discussion.
+If you like ambitious problems: **local web UI**, **graph/context packs**, and **eval harness** are the meatiest next contributions. Open an issue to claim one or kick off a design discussion.
 
-Explicitly not planned (good ideas, just not for this repo): full static-site generator, desktop or mobile apps, fine-tuning, a formal ontology engine, heavy graph reasoning.
+Explicitly not planned (good ideas, just not for this repo): full static-site generator, desktop or mobile apps, fine-tuning, a formal ontology engine, heavy graph database infrastructure.
 
 ## Requirements