@sdsrs/llm-wiki 0.6.0 → 0.6.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +32 -0
- package/README.md +192 -157
- package/bin/llm-wiki.mjs +6 -2
- package/package.json +1 -1
- package/src/export.mjs +60 -2
- package/src/scanner.mjs +31 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,37 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.6.2 (2026-07-11)
|
|
4
|
+
|
|
5
|
+
No breaking changes, no KB migration, no default-behavior change — one additive
|
|
6
|
+
export format.
|
|
7
|
+
|
|
8
|
+
- feat(export): `--format canvas` writes a JSON Canvas 1.0 file
|
|
9
|
+
(`<kb>/graph.canvas`) — a domain map of `file` cards laid out in columns by
|
|
10
|
+
page type (source / entity / concept / comparison / raw), ordered by link
|
|
11
|
+
degree, colored by type, with invalidated pages flagged red. The cards open
|
|
12
|
+
as live, clickable notes in Obsidian Canvas when the KB is opened as its own
|
|
13
|
+
vault. Zero-LLM, alongside the existing graphml / cypher / html / markdown
|
|
14
|
+
exports. Suite 174 → 178.
|
|
15
|
+
|
|
16
|
+
## 0.6.1 (2026-07-11)
|
|
17
|
+
|
|
18
|
+
No breaking changes, no KB migration, no default-behavior change: the only
|
|
19
|
+
functional addition is advisory `scan` output (exit code and existing lines
|
|
20
|
+
untouched).
|
|
21
|
+
|
|
22
|
+
- feat(scan): domain-mixture heuristic — warns when a source directory looks
|
|
23
|
+
multi-domain and suggests one KB per domain. Two zero-LLM signals: language
|
|
24
|
+
mix across scanned text files (minority language ≥3 files and ≥25%) and
|
|
25
|
+
dispersed wiki tags (≥10 tagged pages with no tag covering ≥30%). Advisory
|
|
26
|
+
only; the scan report gains a `domainMixture` field. Suite 171 → 174.
|
|
27
|
+
- docs: README restructured (install → what it does → why-not-RAG →
|
|
28
|
+
usage → FAQ); GitHub repo description refreshed.
|
|
29
|
+
- Measured 2026-07-11 (dogfood KB, 28 probes, `locatePages` 'auto' — the path
|
|
30
|
+
`wiki_search`/skills use): enabling `vectorEnabled` on a KB lifts Recall@5
|
|
31
|
+
0.733 → 0.967 (22/30 → 29/30) and MRR 0.646 → 0.769; the previously missed
|
|
32
|
+
cross-language probe now hits via the vector channel. KB-side config, not a
|
|
33
|
+
code change — recorded here as the reference numbers behind the README claim.
|
|
34
|
+
|
|
3
35
|
## 0.6.0 (2026-07-11)
|
|
4
36
|
|
|
5
37
|
No breaking changes, no KB migration. Default behavior unchanged for KBs
|
package/README.md
CHANGED
|
@@ -1,29 +1,83 @@
|
|
|
1
1
|
# @sdsrs/llm-wiki
|
|
2
2
|
|
|
3
|
-
>
|
|
3
|
+
> Compile a messy folder of documents into a knowledge base your coding agent
|
|
4
|
+
> maintains — and answers from with **whole pages, never chunks**.
|
|
4
5
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
Claude Code / Codex via the bundled skills.
|
|
6
|
+
Point it at PDFs, DOCX, HTML, Markdown. It builds a Karpathy-style `llm_wiki`:
|
|
7
|
+
an immutable `raw/` layer of converted sources plus a `wiki/` layer of full,
|
|
8
|
+
self-contained, cross-linked pages. Query it from the CLI, from Claude Code /
|
|
9
|
+
Codex via bundled skills, or from any MCP client.
|
|
10
10
|
|
|
11
|
-
## Install
|
|
11
|
+
## Install
|
|
12
|
+
|
|
13
|
+
No install needed — run everything with `npx`:
|
|
12
14
|
|
|
13
15
|
```sh
|
|
14
|
-
npx @sdsrs/llm-wiki
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
16
|
+
npx @sdsrs/llm-wiki@0 --help
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Or install globally:
|
|
20
|
+
|
|
21
|
+
```sh
|
|
22
|
+
npm i -g @sdsrs/llm-wiki # installs the `llm-wiki` binary
|
|
20
23
|
```
|
|
21
24
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
+
> Always spell the package `@sdsrs/llm-wiki` — the bare `llm-wiki` name on npm
|
|
26
|
+
> is an unrelated third-party package.
|
|
27
|
+
|
|
28
|
+
**Quickstart:**
|
|
29
|
+
|
|
30
|
+
```sh
|
|
31
|
+
npx @sdsrs/llm-wiki@0 init my-kb # scaffold raw/, wiki/, AGENTS.md, wiki.config.json
|
|
32
|
+
cd my-kb
|
|
33
|
+
npx @sdsrs/llm-wiki@0 scan ~/Documents/src # inventory: dedup, batches, token estimate
|
|
34
|
+
npx @sdsrs/llm-wiki@0 convert # convert planned files into raw/*.md
|
|
35
|
+
# build the wiki/ pages with the wiki-build skill (Claude Code / Codex — see "Skills" below)
|
|
36
|
+
npx @sdsrs/llm-wiki@0 ask "what did we decide about X?"
|
|
37
|
+
```
|
|
25
38
|
|
|
26
|
-
##
|
|
39
|
+
## What it does
|
|
40
|
+
|
|
41
|
+
- **Compiles** documents into a two-layer KB: `raw/` (immutable converted
|
|
42
|
+
markdown, agents only read it) and `wiki/` (typed pages — sources, entities,
|
|
43
|
+
concepts, comparisons — written and maintained by an agent following the KB's
|
|
44
|
+
`AGENTS.md` contract).
|
|
45
|
+
- **Answers questions** with citations: retrieval locates pages, then the model
|
|
46
|
+
reads them *whole*. When selected pages exceed the token budget, the
|
|
47
|
+
lowest-ranked pages are dropped entirely — never truncated mid-page.
|
|
48
|
+
- **Keeps itself honest**: `lint` produces mechanical checks plus a semantic
|
|
49
|
+
worklist; `status` tracks what changed upstream; outdated pages are marked
|
|
50
|
+
`status: invalidated` (with `superseded_by`), never deleted.
|
|
51
|
+
- **Stays cheap**: the CLI calls an LLM only for `ask` (and `embed` if you opt
|
|
52
|
+
into vectors). Scanning, converting, indexing, linting, and graph queries are
|
|
53
|
+
all zero-LLM.
|
|
54
|
+
|
|
55
|
+
## Why this instead of RAG?
|
|
56
|
+
|
|
57
|
+
- **Compile once, stay fresh** — instead of re-interpreting raw documents on
|
|
58
|
+
every query, knowledge is compiled into curated pages once and kept fresh
|
|
59
|
+
incrementally.
|
|
60
|
+
- **Whole pages, never chunks** — the model always reads coherent,
|
|
61
|
+
self-contained documents, so answers come with real provenance instead of
|
|
62
|
+
stitched fragments.
|
|
63
|
+
- **No vector DB required** — retrieval is BM25 out of the box; optional
|
|
64
|
+
whole-page embeddings live in one sidecar JSON file. On our dogfood KB
|
|
65
|
+
(28 probes), enabling vectors lifted Recall@5 from 0.73 to 0.97 — the gap
|
|
66
|
+
was almost entirely cross-language queries.
|
|
67
|
+
- **A real link graph, queried without an LLM** — `graph path | neighbors |
|
|
68
|
+
hubs` traverse wikilinks and typed relations (`implements`, `supersedes`, …)
|
|
69
|
+
instantly.
|
|
70
|
+
- **Agent-native, tool-agnostic** — the same KB serves a standalone CLI,
|
|
71
|
+
Claude Code / Codex skills, an MCP server (Cursor, Windsurf, …), and opens
|
|
72
|
+
directly as an Obsidian vault.
|
|
73
|
+
- **Prompt-injection posture built in** — page content is treated as untrusted
|
|
74
|
+
data everywhere a model sees it, with an explicit notice.
|
|
75
|
+
- **Nothing is ever lost** — `raw/` is immutable; wiki knowledge is
|
|
76
|
+
invalidated, not deleted, so decisions keep their history.
|
|
77
|
+
|
|
78
|
+
## Usage
|
|
79
|
+
|
|
80
|
+
### Commands
|
|
27
81
|
|
|
28
82
|
| command | purpose |
|
|
29
83
|
|---|---|
|
|
@@ -34,59 +88,52 @@ CLI does not call an LLM to build pages, only to `ask`.
|
|
|
34
88
|
| `ask <question>` | answer from the KB using full pages (never chunks), with citations |
|
|
35
89
|
| `lint` | mechanical checks + a semantic worklist for the agent |
|
|
36
90
|
| `status` | incremental state: uncompiled raw files, source-dir diff, affected wiki pages |
|
|
37
|
-
| `export` | export the wiki graph as GraphML, Cypher, or a self-contained interactive HTML viewer |
|
|
38
91
|
| `graph` | query `graph.json` with `path` / `neighbors` / `hubs` (zero-LLM traversal) |
|
|
39
|
-
| `embed` | compute/update page embeddings (wiki/.vectors.json) for optional vector
|
|
40
|
-
| `
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
pages
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
92
|
+
| `embed` | compute/update page embeddings (`wiki/.vectors.json`) for optional vector location |
|
|
93
|
+
| `export` | export the graph as GraphML, Cypher, JSON Canvas, or an interactive HTML viewer; or the wiki as standard-markdown copies |
|
|
94
|
+
| `mcp` | run a read-only MCP server (stdio) over the KB |
|
|
95
|
+
|
|
96
|
+
All commands take `--kb <dir>` (default `.`). `ask` supports `-k <n>` (pages to
|
|
97
|
+
load) and `--retrieve-only` (locate pages without calling the LLM).
|
|
98
|
+
|
|
99
|
+
`scan` also warns when the source looks multi-domain (mixed-language files or
|
|
100
|
+
dispersed wiki tags) and suggests splitting — one KB per domain.
|
|
101
|
+
|
|
102
|
+
### Asking questions
|
|
103
|
+
|
|
104
|
+
Retrieval is lexical (BM25) by default. When it finds nothing — typically a
|
|
105
|
+
question asked in a different language than the KB pages, or fully rephrased —
|
|
106
|
+
`ask` falls back to letting the model pick pages from the KB listing
|
|
107
|
+
(`llms.txt`); only ids of real, non-invalidated pages are accepted. Pages are
|
|
108
|
+
always sent whole; over `askTokenBudget` (`wiki.config.json`, default 32000)
|
|
109
|
+
the lowest-ranked pages are dropped, never truncated.
|
|
110
|
+
|
|
111
|
+
### Vector page location (optional)
|
|
112
|
+
|
|
113
|
+
Fixes cross-language and rephrased queries. Three steps:
|
|
114
|
+
|
|
115
|
+
1. Set `"vectorEnabled": true` in `wiki.config.json`.
|
|
116
|
+
2. Add `"embeddingModel"` to your provider in `~/.llm-wiki/config.json`.
|
|
117
|
+
3. Run `npx @sdsrs/llm-wiki@0 embed`.
|
|
118
|
+
|
|
119
|
+
`ask` (and the MCP `wiki_search` tool) then fuse BM25 with whole-page cosine
|
|
120
|
+
similarity (RRF); `--retrieve-only` labels each hit `[bm25]`, `[vector]` or
|
|
121
|
+
`[bm25+vector]`. Vectors only *locate* pages — context is still whole pages,
|
|
61
122
|
never chunks. Any vector-path failure falls back to BM25 with a warning.
|
|
62
123
|
|
|
63
124
|
### Graph queries
|
|
64
125
|
|
|
65
126
|
`graph.json` (rebuilt by `index`) is a link graph over the pages — wikilinks,
|
|
66
|
-
`superseded_by`, and typed relations
|
|
127
|
+
`superseded_by`, and typed relations:
|
|
67
128
|
|
|
68
129
|
```sh
|
|
69
|
-
npx @sdsrs/llm-wiki@0 graph hubs --kb ./kb --top 5
|
|
130
|
+
npx @sdsrs/llm-wiki@0 graph hubs --kb ./kb --top 5 # most-connected pages
|
|
70
131
|
npx @sdsrs/llm-wiki@0 graph path <from-id> <to-id> --kb ./kb # shortest link chain
|
|
71
132
|
npx @sdsrs/llm-wiki@0 graph neighbors <id> -d 2 --kb ./kb # pages within N hops
|
|
72
133
|
```
|
|
73
134
|
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
```
|
|
77
|
-
31 concepts/llm-wiki (in 10 / out 21) LLM Wiki
|
|
78
|
-
31 entities/karpathy (in 18 / out 13) Andrej Karpathy
|
|
79
|
-
21 entities/obsidian (in 10 / out 11) Obsidian
|
|
80
|
-
16 concepts/rag (in 4 / out 12) RAG（检索增强生成）
|
|
81
|
-
13 concepts/ingest-query-lint (in 3 / out 10) Ingest / Query / Lint 三大操作
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
The same three queries are exposed to MCP agents as `wiki_graph`
|
|
85
|
-
(`op: path | neighbors | hubs`).
|
|
86
|
-
|
|
87
|
-
**Typed relations.** Body `[[wikilinks]]` capture that two pages relate; when the
|
|
88
|
-
*kind* of link matters, record it in the page frontmatter (edges merge into
|
|
89
|
-
`graph.json` on `index`):
|
|
135
|
+
When the *kind* of link matters, record it in page frontmatter (edges merge
|
|
136
|
+
into `graph.json` on `index`):
|
|
90
137
|
|
|
91
138
|
```yaml
|
|
92
139
|
relations:
|
|
@@ -95,124 +142,71 @@ relations:
|
|
|
95
142
|
confidence: inferred # extracted | inferred | ambiguous
|
|
96
143
|
```
|
|
97
144
|
|
|
98
|
-
`
|
|
99
|
-
|
|
100
|
-
CLI-derived structure (`source` / `superseded_by` edges), `inferred` = agent
|
|
101
|
-
judgment (the default, also carried by wikilinks), `ambiguous` = flagged for a
|
|
102
|
-
human to resolve. `llm-wiki lint` validates that each relation target exists and
|
|
103
|
-
its `type` is in the vocabulary, and counts relation targets as incoming links for
|
|
104
|
-
orphan detection.
|
|
105
|
-
|
|
106
|
-
## Obsidian integration
|
|
107
|
-
|
|
108
|
-
An llm_wiki KB is a plain folder of markdown with YAML frontmatter, so it opens
|
|
109
|
-
directly as an Obsidian vault — "Open folder as vault", point it at `./kb`. Don't
|
|
110
|
-
pre-create or edit `.obsidian/` — let Obsidian create and manage its own config
|
|
111
|
-
folder; llm-wiki never writes it.
|
|
112
|
-
|
|
113
|
-
What lights up out of the box:
|
|
114
|
-
|
|
115
|
-
- **Graph view + backlinks** from the path-style `[[wikilinks]]` the pages already
|
|
116
|
-
use (`[[entities/karpathy]]`) — they resolve across the `sources/`, `entities/`,
|
|
117
|
-
`concepts/`, `comparisons/` subfolders and populate the graph and backlinks pane.
|
|
118
|
-
- **Properties panel** from the page frontmatter (`type`, `title`, `tags`, `created`,
|
|
119
|
-
`updated`, …). `tags` and `aliases` are YAML string lists, exactly Obsidian's
|
|
120
|
-
expected shape (no `#` prefix inside the frontmatter list).
|
|
121
|
-
- **Bases** (a core plugin) reads those properties into table/card views you can
|
|
122
|
-
filter on `type`, `tags`, or `status` — e.g. a base over `type == "concept"` with a
|
|
123
|
-
filter `status != "invalidated"` to hide retired pages.
|
|
124
|
-
- **Link previews**: the frontmatter `description` doubles as Obsidian's hover-preview
|
|
125
|
-
text for a page.
|
|
126
|
-
|
|
127
|
-
Optional per-page `aliases` (a YAML list of alternative names) feed Obsidian's link
|
|
128
|
-
autocomplete and alternate-name resolution; add them at page creation for topics with
|
|
129
|
-
well-known synonyms, translations, or abbreviations.
|
|
130
|
-
|
|
131
|
-
Typed `relations` (a list of objects — `to`/`type`/`confidence`) are valid YAML and
|
|
132
|
-
round-trip fine, but a list-of-objects renders best in source mode rather than the
|
|
133
|
-
Properties panel; they are primarily consumed by `llm-wiki graph` / the MCP server, not
|
|
134
|
-
eyeballed in Obsidian.
|
|
135
|
-
|
|
136
|
-
**Positioning.** Obsidian is for browsing and annotation; agents (via the wiki-* skills)
|
|
137
|
-
remain the writers of `wiki/` pages. If you edit pages by hand in Obsidian, run
|
|
138
|
-
`npx @sdsrs/llm-wiki@0 index --kb <kb>` afterwards to rebuild `index.md`, `graph.json`,
|
|
139
|
-
and `llms.txt`. There is deliberately no bidirectional sync: concurrent writes to the
|
|
140
|
-
same file (agent and editor at once) can corrupt content silently, which is why it stays
|
|
141
|
-
out of scope. For tools that need standard markdown links instead of wikilinks, export a
|
|
142
|
-
converted copy: `npx @sdsrs/llm-wiki@0 export --format markdown --kb <kb>`. The default
|
|
143
|
-
output dir is `<kb>/wiki-md/` (inside the KB so its `../raw/...` links resolve against the
|
|
144
|
-
real raw layer); if you use the KB itself as an Obsidian vault, add `wiki-md/` to Obsidian's
|
|
145
|
-
"Excluded files" so the converted copies aren't indexed as duplicate notes — or export with
|
|
146
|
-
`--out` outside the KB, at the cost of `raw/` links not resolving.
|
|
147
|
-
|
|
148
|
-
**Verification checklist.** The conventions above are derived from Obsidian's docs, not
|
|
149
|
-
tested against a specific Obsidian build. Run this once after opening the vault to confirm
|
|
150
|
-
them on your Obsidian version:
|
|
151
|
-
|
|
152
|
-
- [ ] Open `./kb` as a vault ("Open folder as vault").
|
|
153
|
-
- [ ] Graph view shows clusters of typed pages linked by wikilinks.
|
|
154
|
-
- [ ] Open a page — the backlinks panel is non-empty for a linked page.
|
|
155
|
-
- [ ] Create a Base filtered to `type == "concept"`.
|
|
156
|
-
- [ ] Add the filter `status != "invalidated"` and confirm invalidated pages drop out.
|
|
157
|
-
|
|
158
|
-
## LLM config
|
|
159
|
-
|
|
160
|
-
`ask` needs an OpenAI-compatible endpoint. Configure `~/.llm-wiki/config.json`:
|
|
161
|
-
|
|
162
|
-
```json
|
|
163
|
-
{
|
|
164
|
-
"priority": ["openai", "openrouter"],
|
|
165
|
-
"providers": {
|
|
166
|
-
"openai": { "baseURL": "https://api.openai.com/v1", "apiKeyEnv": "OPENAI_API_KEY", "model": "gpt-4o-mini" },
|
|
167
|
-
"openrouter": { "baseURL": "https://openrouter.ai/api/v1", "apiKeyEnv": "OPENROUTER_API_KEY", "model": "anthropic/claude-sonnet-5" }
|
|
168
|
-
}
|
|
169
|
-
}
|
|
170
|
-
```
|
|
145
|
+
`lint` validates each relation target and type; `ambiguous` marks edges for a
|
|
146
|
+
human to resolve.
|
|
171
147
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
`HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars.
|
|
148
|
+
`export --format canvas` writes a JSON Canvas (`graph.canvas`) domain map —
|
|
149
|
+
`file` cards laid out by page type that open as live notes in Obsidian Canvas.
|
|
175
150
|
|
|
176
|
-
|
|
151
|
+
### Skills (Claude Code / Codex)
|
|
177
152
|
|
|
178
|
-
The skills (`wiki-build`, `wiki-ingest`, `wiki-query`, `wiki-lint`,
|
|
179
|
-
`wiki-distill`) let a coding agent build and maintain the KB
|
|
153
|
+
The bundled skills (`wiki-build`, `wiki-ingest`, `wiki-query`, `wiki-lint`,
|
|
154
|
+
`wiki-connect`, `wiki-distill`) let a coding agent build and maintain the KB —
|
|
155
|
+
the CLI never LLM-writes pages itself.
|
|
180
156
|
|
|
181
|
-
|
|
157
|
+
As a Claude Code plugin (recommended — updates with the repo):
|
|
182
158
|
|
|
183
159
|
```
|
|
184
160
|
/plugin marketplace add sdsrss/llm_wiki
|
|
185
161
|
/plugin install llm-wiki
|
|
186
162
|
```
|
|
187
163
|
|
|
188
|
-
|
|
164
|
+
Or copied into a project (works for Codex and other agents too):
|
|
189
165
|
|
|
190
166
|
```sh
|
|
191
|
-
npx @sdsrs/llm-wiki install-skills # copy wiki-* skills into ./.claude
|
|
192
|
-
npx @sdsrs/llm-wiki connect <projectDir> --kb <path> # register
|
|
167
|
+
npx @sdsrs/llm-wiki@0 install-skills # copy wiki-* skills into ./.claude
|
|
168
|
+
npx @sdsrs/llm-wiki@0 connect <projectDir> --kb <path> # register the KB in a project's CLAUDE.md
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### MCP server (Cursor, Windsurf, …)
|
|
172
|
+
|
|
173
|
+
```json
|
|
174
|
+
{ "mcpServers": { "my-kb": { "command": "npx", "args": ["-y", "@sdsrs/llm-wiki@0", "mcp", "--kb", "/path/to/my-kb"] } } }
|
|
193
175
|
```
|
|
194
176
|
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
177
|
+
Five read-only tools: `wiki_overview`, `wiki_search` (page locator — ids +
|
|
178
|
+
descriptions, never full text), `wiki_read_page`, `wiki_ask` (needs LLM
|
|
179
|
+
config), `wiki_graph`. Page content is served as untrusted data with an
|
|
180
|
+
explicit notice.
|
|
181
|
+
|
|
182
|
+
### Obsidian
|
|
183
|
+
|
|
184
|
+
A KB is a plain folder of markdown with YAML frontmatter — open it directly
|
|
185
|
+
with "Open folder as vault". Graph view and backlinks light up from the
|
|
186
|
+
path-style `[[wikilinks]]`, the Properties panel from the frontmatter, and
|
|
187
|
+
Bases can filter on `type` / `tags` / `status` (e.g. hide
|
|
188
|
+
`status == "invalidated"`). Obsidian is for browsing and annotation; agents
|
|
189
|
+
remain the writers. If you hand-edit pages, run
|
|
190
|
+
`npx @sdsrs/llm-wiki@0 index` afterwards to rebuild the derived files.
|
|
199
191
|
|
|
200
|
-
|
|
192
|
+
### LLM config
|
|
201
193
|
|
|
202
|
-
|
|
194
|
+
`ask` needs an OpenAI-compatible endpoint. Configure `~/.llm-wiki/config.json`:
|
|
203
195
|
|
|
204
196
|
```json
|
|
205
|
-
{
|
|
197
|
+
{
|
|
198
|
+
"priority": ["openai", "openrouter"],
|
|
199
|
+
"providers": {
|
|
200
|
+
"openai": { "baseURL": "https://api.openai.com/v1", "apiKeyEnv": "OPENAI_API_KEY", "model": "gpt-4o-mini" },
|
|
201
|
+
"openrouter": { "baseURL": "https://openrouter.ai/api/v1", "apiKeyEnv": "OPENROUTER_API_KEY", "model": "anthropic/claude-sonnet-5" }
|
|
202
|
+
}
|
|
203
|
+
}
|
|
206
204
|
```
|
|
207
205
|
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
by id), `wiki_ask` (one-shot Q&A with citations; needs `~/.llm-wiki/config.json`,
|
|
211
|
-
errors with guidance otherwise), `wiki_graph` (zero-LLM link-graph queries —
|
|
212
|
-
`op: path | neighbors | hubs` over `graph.json`). Page content is served as untrusted data with an
|
|
213
|
-
explicit notice, mirroring the skills' prompt-injection posture.
|
|
206
|
+
The first provider whose `apiKeyEnv` env var is set wins. Proxies are honored
|
|
207
|
+
via the standard `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars.
|
|
214
208
|
|
|
215
|
-
|
|
209
|
+
### KB layout
|
|
216
210
|
|
|
217
211
|
```
|
|
218
212
|
my-kb/
|
|
@@ -224,10 +218,51 @@ my-kb/
|
|
|
224
218
|
wiki.config.json thresholds, batch size, language
|
|
225
219
|
```
|
|
226
220
|
|
|
221
|
+
## FAQ
|
|
222
|
+
|
|
223
|
+
**Do I need an API key?**
|
|
224
|
+
Only for `ask` (answering) and `embed` (optional vectors). Everything else —
|
|
225
|
+
scan, convert, index, lint, graph, export, the skills' bookkeeping — is
|
|
226
|
+
zero-LLM and offline.
|
|
227
|
+
|
|
228
|
+
**Am I locked in?**
|
|
229
|
+
No. A KB is plain markdown + YAML frontmatter in a folder. It opens as an
|
|
230
|
+
Obsidian vault as-is, and `export --format markdown` produces a copy with
|
|
231
|
+
standard links for tools that don't speak wikilinks.
|
|
232
|
+
|
|
233
|
+
**Can I edit wiki pages by hand?**
|
|
234
|
+
Yes, but run `index` afterwards to rebuild `index.md` / `graph.json` /
|
|
235
|
+
`llms.txt`. There is deliberately no editor⇄agent bidirectional sync:
|
|
236
|
+
concurrent writes to the same file can corrupt content silently.
|
|
237
|
+
|
|
238
|
+
**What happens to outdated knowledge?**
|
|
239
|
+
It's never deleted. Pages get `status: invalidated` (optionally
|
|
240
|
+
`superseded_by: <new-page>`) and drop out of `llms.txt` and retrieval, while
|
|
241
|
+
their history stays browsable.
|
|
242
|
+
|
|
243
|
+
**Can one KB hold several domains?**
|
|
244
|
+
One KB per domain works best. `scan` warns when a source directory looks
|
|
245
|
+
multi-domain (mixed languages, dispersed tags) and suggests splitting.
|
|
246
|
+
|
|
247
|
+
**Why do the docs write `@sdsrs/llm-wiki@0` everywhere?**
|
|
248
|
+
The bare `llm-wiki` npm name belongs to an unrelated package, and pinning the
|
|
249
|
+
major (`@0`) keeps `npx` runs reproducible. If a project's `CLAUDE.md` still
|
|
250
|
+
contains a pre-0.2.0 `npx llm-wiki ...` block, re-run `connect` once — the
|
|
251
|
+
block is rewritten in place, idempotently.
|
|
252
|
+
|
|
253
|
+
**Is page content trusted?**
|
|
254
|
+
No. Source documents and wiki pages are treated as untrusted data in every
|
|
255
|
+
model-facing prompt (ask, MCP, skills), with an explicit
|
|
256
|
+
never-follow-instructions notice — a prompt-injection guardrail, not an
|
|
257
|
+
afterthought.
|
|
258
|
+
|
|
227
259
|
## Iron rules
|
|
228
260
|
|
|
229
261
|
- `raw/` is immutable — agents read it, never modify it.
|
|
230
262
|
- Pages are full and self-contained; retrieval and `ask` load whole pages, never chunks.
|
|
231
263
|
- Source documents are untrusted: page content is data, never instructions to follow.
|
|
232
|
-
- Knowledge is never deleted
|
|
233
|
-
|
|
264
|
+
- Knowledge is never deleted — invalidate (and supersede), don't erase.
|
|
265
|
+
|
|
266
|
+
## License
|
|
267
|
+
|
|
268
|
+
MIT
|
package/bin/llm-wiki.mjs
CHANGED
|
@@ -31,6 +31,10 @@ program.command('scan <srcDir>')
|
|
|
31
31
|
console.log(`files: ${r.files.length} (skipped ${r.skipped.length})`)
|
|
32
32
|
console.log(`duplicates: ${r.duplicates.exact.length} exact, ${r.duplicates.near.length} near`)
|
|
33
33
|
console.log(`incremental: +${r.incremental.added} ~${r.incremental.changed} -${r.incremental.removed} =${r.incremental.unchanged}`)
|
|
34
|
+
const dm = r.domainMixture
|
|
35
|
+
if (dm.language.flagged) console.log(`warning: mixed-language source (zh ${dm.language.zh} / en ${dm.language.en} files) — this KB may span multiple domains`)
|
|
36
|
+
if (dm.tags?.flagged) console.log(`warning: wiki tags are dispersed (top tag covers ${Math.round(dm.tags.topShare * 100)}% of ${dm.tags.pages} pages) — this KB may span multiple domains`)
|
|
37
|
+
if (dm.flagged) console.log('hint: llm-wiki works best with one domain per KB; consider splitting the source into separate KBs')
|
|
34
38
|
console.log(`compile plan: ${r.batches.length} batches -> ~${r.estimate.inputTokens} in / ~${r.estimate.outputTokens} out tokens`)
|
|
35
39
|
console.log(`plan saved to ${opts.kb}/.scan-plan.json`)
|
|
36
40
|
})
|
|
@@ -124,9 +128,9 @@ program.command('install-skills')
|
|
|
124
128
|
})
|
|
125
129
|
|
|
126
130
|
program.command('export')
|
|
127
|
-
.description('export wiki/graph.json (graphml | cypher | html) or a markdown-links copy of the wiki (markdown)')
|
|
131
|
+
.description('export wiki/graph.json (graphml | cypher | html | canvas) or a markdown-links copy of the wiki (markdown)')
|
|
128
132
|
.option('--kb <dir>', 'knowledge base root', '.')
|
|
129
|
-
.requiredOption('--format <format>', 'graphml | cypher | html | markdown')
|
|
133
|
+
.requiredOption('--format <format>', 'graphml | cypher | html | canvas | markdown')
|
|
130
134
|
.option('--out <file>', 'output path (default: <kb>/graph.<ext>, or <kb>/wiki-md/ for markdown)')
|
|
131
135
|
.action((opts) => {
|
|
132
136
|
if (opts.format === 'markdown') {
|
package/package.json
CHANGED
package/src/export.mjs
CHANGED
|
@@ -236,13 +236,71 @@ step()
|
|
|
236
236
|
`
|
|
237
237
|
}
|
|
238
238
|
|
|
239
|
-
|
|
239
|
+
// JSON Canvas 1.0 (jsoncanvas.org) — an open node/edge format Obsidian Canvas and
|
|
240
|
+
// other tools read. Nodes are `file` cards pointing at the real KB pages
|
|
241
|
+
// (vault-relative from the KB root, where the .canvas is written), so opening the
|
|
242
|
+
// map in Obsidian gives live, clickable notes. Layout is deterministic — one
|
|
243
|
+
// column per node type, ordered by degree — so exports are stable and diffable.
|
|
244
|
+
const CANVAS_COLORS = { source: '5', entity: '4', concept: '6', comparison: '2' } // JSON Canvas preset ids; raw/unknown → default gray
|
|
245
|
+
const CANVAS_TYPE_ORDER = ['source', 'entity', 'concept', 'comparison', 'raw']
|
|
246
|
+
const CANVAS_NODE_W = 260
|
|
247
|
+
const CANVAS_NODE_H = 60
|
|
248
|
+
const CANVAS_COL_GAP = 340
|
|
249
|
+
const CANVAS_ROW_GAP = 90
|
|
250
|
+
|
|
251
|
+
export function toCanvas(graph) {
|
|
252
|
+
const degree = new Map()
|
|
253
|
+
for (const e of graph.edges) for (const end of [e.source, e.target]) degree.set(end, (degree.get(end) ?? 0) + 1)
|
|
254
|
+
|
|
255
|
+
const cols = new Map()
|
|
256
|
+
for (const n of graph.nodes) {
|
|
257
|
+
const key = CANVAS_TYPE_ORDER.includes(n.type) ? n.type : 'raw'
|
|
258
|
+
if (!cols.has(key)) cols.set(key, [])
|
|
259
|
+
cols.get(key).push(n)
|
|
260
|
+
}
|
|
261
|
+
const rank = (t) => { const i = CANVAS_TYPE_ORDER.indexOf(t); return i === -1 ? CANVAS_TYPE_ORDER.length : i }
|
|
262
|
+
const colKeys = [...cols.keys()].sort((a, b) => rank(a) - rank(b))
|
|
263
|
+
|
|
264
|
+
const nodes = []
|
|
265
|
+
colKeys.forEach((key, ci) => {
|
|
266
|
+
const column = cols.get(key).sort((a, b) =>
|
|
267
|
+
(degree.get(b.id) ?? 0) - (degree.get(a.id) ?? 0) || a.id.localeCompare(b.id))
|
|
268
|
+
column.forEach((n, ri) => {
|
|
269
|
+
const node = {
|
|
270
|
+
id: n.id,
|
|
271
|
+
type: 'file',
|
|
272
|
+
file: n.id.startsWith('raw/') ? n.id : `wiki/${n.id}.md`,
|
|
273
|
+
x: ci * CANVAS_COL_GAP,
|
|
274
|
+
y: ri * CANVAS_ROW_GAP,
|
|
275
|
+
width: CANVAS_NODE_W,
|
|
276
|
+
height: CANVAS_NODE_H,
|
|
277
|
+
}
|
|
278
|
+
const color = n.status === 'invalidated' ? '1' : CANVAS_COLORS[n.type]
|
|
279
|
+
if (color) node.color = color
|
|
280
|
+
nodes.push(node)
|
|
281
|
+
})
|
|
282
|
+
})
|
|
283
|
+
|
|
284
|
+
const present = new Set(nodes.map(n => n.id))
|
|
285
|
+
const edges = graph.edges
|
|
286
|
+
.filter(e => present.has(e.source) && present.has(e.target))
|
|
287
|
+
.map((e, i) => {
|
|
288
|
+
const edge = { id: `e${i}`, fromNode: e.source, toNode: e.target }
|
|
289
|
+
if (e.type) edge.label = e.type
|
|
290
|
+
return edge
|
|
291
|
+
})
|
|
292
|
+
|
|
293
|
+
return JSON.stringify({ nodes, edges }, null, 2) + '\n'
|
|
294
|
+
}
|
|
295
|
+
|
|
296
|
+
const RENDERERS = { graphml: toGraphML, cypher: toCypher, html: toHtml, canvas: toCanvas }
|
|
297
|
+
const EXT = { graphml: 'graphml', cypher: 'cypher', html: 'html', canvas: 'canvas' }
|
|
240
298
|
|
|
241
299
|
export function exportGraph(kbRoot, { format, out } = {}) {
|
|
242
300
|
const render = RENDERERS[format]
|
|
243
301
|
if (!render) throw new Error(`unknown format: ${format} (expected ${Object.keys(RENDERERS).join(' | ')} | markdown)`)
|
|
244
302
|
const graph = loadGraph(kbRoot)
|
|
245
|
-
const outPath = out ?? path.join(kbRoot, `graph.${format
|
|
303
|
+
const outPath = out ?? path.join(kbRoot, `graph.${EXT[format]}`)
|
|
246
304
|
fs.writeFileSync(outPath, render(graph))
|
|
247
305
|
return { out: outPath, nodeCount: graph.nodes.length, edgeCount: graph.edges.length }
|
|
248
306
|
}
|
package/src/scanner.mjs
CHANGED
|
@@ -5,9 +5,39 @@ import { loadKbConfig } from './templates.mjs'
|
|
|
5
5
|
import { SUPPORTED_EXTS } from './convert.mjs'
|
|
6
6
|
import { sha256File, minhashSignature, jaccardEstimate } from './hashing.mjs'
|
|
7
7
|
import { loadManifest, diffManifest } from './manifest.mjs'
|
|
8
|
+
import { listWikiPages, isInvalidated } from './pages.mjs'
|
|
8
9
|
|
|
9
10
|
const TEXT_EXTS = ['.md', '.markdown', '.txt', '.html', '.htm']
|
|
10
11
|
const NEAR_DUP_THRESHOLD = 0.85
|
|
12
|
+
const LANG_MIX_MIN_FILES = 3
|
|
13
|
+
const LANG_MIX_MINORITY_SHARE = 0.25
|
|
14
|
+
const TAG_MIN_PAGES = 10
|
|
15
|
+
const TAG_TOP_SHARE_MIN = 0.3
|
|
16
|
+
|
|
17
|
+
// Advisory guardrail for the one-KB-per-domain rule, from signals scan already
|
|
18
|
+
// has: language mix across scanned text files, and tag dispersion across the
|
|
19
|
+
// built wiki (a cohesive KB has at least one tag most pages share). No LLM,
|
|
20
|
+
// no network; nothing blocks on it.
|
|
21
|
+
function detectDomainMixture(files, kbRoot) {
|
|
22
|
+
const zh = files.filter(f => f.lang === 'zh').length
|
|
23
|
+
const en = files.filter(f => f.lang === 'en').length
|
|
24
|
+
const minority = Math.min(zh, en)
|
|
25
|
+
const language = {
|
|
26
|
+
zh,
|
|
27
|
+
en,
|
|
28
|
+
flagged: minority >= LANG_MIX_MIN_FILES && minority / (zh + en) >= LANG_MIX_MINORITY_SHARE,
|
|
29
|
+
}
|
|
30
|
+
let tags = null
|
|
31
|
+
const pages = listWikiPages(kbRoot).filter(p =>
|
|
32
|
+
!p.error && !isInvalidated(p) && Array.isArray(p.data.tags) && p.data.tags.length > 0)
|
|
33
|
+
if (pages.length >= TAG_MIN_PAGES) {
|
|
34
|
+
const counts = new Map()
|
|
35
|
+
for (const p of pages) for (const t of new Set(p.data.tags)) counts.set(t, (counts.get(t) ?? 0) + 1)
|
|
36
|
+
const topShare = Math.max(...counts.values()) / pages.length
|
|
37
|
+
tags = { pages: pages.length, distinct: counts.size, topShare: Number(topShare.toFixed(2)), flagged: topShare < TAG_TOP_SHARE_MIN }
|
|
38
|
+
}
|
|
39
|
+
return { language, tags, flagged: language.flagged || (tags?.flagged ?? false) }
|
|
40
|
+
}
|
|
11
41
|
|
|
12
42
|
export function estimateTokens(text) {
|
|
13
43
|
let cjk = 0
|
|
@@ -103,6 +133,7 @@ export async function scanSource(srcDir, kbRoot, { exclude = [], persist = true
|
|
|
103
133
|
skipped,
|
|
104
134
|
duplicates: { exact, near },
|
|
105
135
|
incremental: { added: diff.added.length, changed: diff.changed.length, removed: diff.removed.length, unchanged: diff.unchanged.length },
|
|
136
|
+
domainMixture: detectDomainMixture(files, kbRoot),
|
|
106
137
|
batches,
|
|
107
138
|
estimate,
|
|
108
139
|
}
|