@sdsrs/llm-wiki 0.6.0 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. package/CHANGELOG.md +19 -0
  2. package/README.md +190 -158
  3. package/bin/llm-wiki.mjs +4 -0
  4. package/package.json +1 -1
  5. package/src/scanner.mjs +31 -0
package/CHANGELOG.md CHANGED
@@ -1,5 +1,24 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.6.1 (2026-07-11)
4
+
5
+ No breaking changes, no KB migration, no default-behavior change: the only
6
+ functional addition is advisory `scan` output (exit code and existing lines
7
+ untouched).
8
+
9
+ - feat(scan): domain-mixture heuristic — warns when a source directory looks
10
+ multi-domain and suggests one KB per domain. Two zero-LLM signals: language
11
+ mix across scanned text files (minority language ≥3 files and ≥25%) and
12
+ dispersed wiki tags (≥10 tagged pages with no tag covering ≥30%). Advisory
13
+ only; the scan report gains a `domainMixture` field. Suite 171 → 174.
14
+ - docs: README restructured (install → what it does → why-not-RAG →
15
+ usage → FAQ); GitHub repo description refreshed.
16
+ - Measured 2026-07-11 (dogfood KB, 28 probes, `locatePages` 'auto' — the path
17
+ `wiki_search`/skills use): enabling `vectorEnabled` on a KB lifts Recall@5
18
+ 0.733 → 0.967 (22/30 → 29/30) and MRR 0.646 → 0.769; the previously missed
19
+ cross-language probe now hits via the vector channel. KB-side config, not a
20
+ code change — recorded here as the reference numbers behind the README claim.
21
+
3
22
  ## 0.6.0 (2026-07-11)
4
23
 
5
24
  No breaking changes, no KB migration. Default behavior unchanged for KBs
package/README.md CHANGED
@@ -1,29 +1,83 @@
1
1
  # @sdsrs/llm-wiki
2
2
 
3
- > npm package `@sdsrs/llm-wiki`; the installed binary is `llm-wiki`.
3
+ > Compile a messy folder of documents into a knowledge base your coding agent
4
+ > maintains — and answers from with **whole pages, never chunks**.
4
5
 
5
- Compile a messy directory of documents (PDF, DOCX, HTML, Markdown, ...) into a
6
- Karpathy-style `llm_wiki` knowledge base: an immutable `raw/` layer of converted
7
- source markdown plus a `wiki/` layer of full, self-contained, cross-linked pages
8
- that a coding agent maintains. Query it standalone from the CLI, or wire it into
9
- Claude Code / Codex via the bundled skills.
6
+ Point it at PDFs, DOCX, HTML, Markdown. It builds a Karpathy-style `llm_wiki`:
7
+ an immutable `raw/` layer of converted sources plus a `wiki/` layer of full,
8
+ self-contained, cross-linked pages. Query it from the CLI, from Claude Code /
9
+ Codex via bundled skills, or from any MCP client.
10
10
 
11
- ## Install / quickstart
11
+ ## Install
12
+
13
+ No install needed — run everything with `npx`:
12
14
 
13
15
  ```sh
14
- npx @sdsrs/llm-wiki init my-kb # scaffold raw/, wiki/, AGENTS.md, wiki.config.json
15
- cd my-kb
16
- npx @sdsrs/llm-wiki scan ~/Documents/src # inventory: dedup, batches, token estimate
17
- npx @sdsrs/llm-wiki convert # convert planned files into raw/*.md
18
- # build the wiki/ pages from raw/ with the wiki-build skill (Claude Code / Codex)
19
- npx @sdsrs/llm-wiki ask "what did we decide about X?"
16
+ npx @sdsrs/llm-wiki@0 --help
17
+ ```
18
+
19
+ Or install globally:
20
+
21
+ ```sh
22
+ npm i -g @sdsrs/llm-wiki # installs the `llm-wiki` binary
20
23
  ```
21
24
 
22
- `scan` + `convert` fill `raw/`. The `wiki/` pages themselves are written by an
23
- agent running the **wiki-build** skill against `AGENTS.md` (the KB contract) — the
24
- CLI does not call an LLM to build pages, only to `ask`.
25
+ > Always spell the package `@sdsrs/llm-wiki` — the bare `llm-wiki` name on npm
26
+ > is an unrelated third-party package.
27
+
28
+ **Quickstart:**
29
+
30
+ ```sh
31
+ npx @sdsrs/llm-wiki@0 init my-kb # scaffold raw/, wiki/, AGENTS.md, wiki.config.json
32
+ cd my-kb
33
+ npx @sdsrs/llm-wiki@0 scan ~/Documents/src # inventory: dedup, batches, token estimate
34
+ npx @sdsrs/llm-wiki@0 convert # convert planned files into raw/*.md
35
+ # build the wiki/ pages with the wiki-build skill (Claude Code / Codex — see "Skills" below)
36
+ npx @sdsrs/llm-wiki@0 ask "what did we decide about X?"
37
+ ```
25
38
 
26
- ## Commands
39
+ ## What it does
40
+
41
+ - **Compiles** documents into a two-layer KB: `raw/` (immutable converted
42
+ markdown, agents only read it) and `wiki/` (typed pages — sources, entities,
43
+ concepts, comparisons — written and maintained by an agent following the KB's
44
+ `AGENTS.md` contract).
45
+ - **Answers questions** with citations: retrieval locates pages, then the model
46
+ reads them *whole*. When selected pages exceed the token budget, the
47
+ lowest-ranked pages are dropped entirely — never truncated mid-page.
48
+ - **Keeps itself honest**: `lint` produces mechanical checks plus a semantic
49
+ worklist; `status` tracks what changed upstream; outdated pages are marked
50
+ `status: invalidated` (with `superseded_by`), never deleted.
51
+ - **Stays cheap**: the CLI calls an LLM only for `ask` (and `embed` if you opt
52
+ into vectors). Scanning, converting, indexing, linting, and graph queries are
53
+ all zero-LLM.
54
+
55
+ ## Why this instead of RAG?
56
+
57
+ - **Compile once, stay fresh** — instead of re-interpreting raw documents on
58
+ every query, knowledge is compiled into curated pages once and kept fresh
59
+ incrementally.
60
+ - **Whole pages, never chunks** — the model always reads coherent,
61
+ self-contained documents, so answers come with real provenance instead of
62
+ stitched fragments.
63
+ - **No vector DB required** — retrieval is BM25 out of the box; optional
64
+ whole-page embeddings live in one sidecar JSON file. On our dogfood KB
65
+ (28 probes), enabling vectors lifted Recall@5 from 0.73 to 0.97 — the gap
66
+ was almost entirely cross-language queries.
67
+ - **A real link graph, queried without an LLM** — `graph path | neighbors |
68
+ hubs` traverse wikilinks and typed relations (`implements`, `supersedes`, …)
69
+ instantly.
70
+ - **Agent-native, tool-agnostic** — the same KB serves a standalone CLI,
71
+ Claude Code / Codex skills, an MCP server (Cursor, Windsurf, …), and opens
72
+ directly as an Obsidian vault.
73
+ - **Prompt-injection posture built in** — page content is treated as untrusted
74
+ data everywhere a model sees it, with an explicit notice.
75
+ - **Nothing is ever lost** — `raw/` is immutable; wiki knowledge is
76
+ invalidated, not deleted, so decisions keep their history.
77
+
78
+ ## Usage
79
+
80
+ ### Commands
27
81
 
28
82
  | command | purpose |
29
83
  |---|---|
@@ -34,59 +88,52 @@ CLI does not call an LLM to build pages, only to `ask`.
34
88
  | `ask <question>` | answer from the KB using full pages (never chunks), with citations |
35
89
  | `lint` | mechanical checks + a semantic worklist for the agent |
36
90
  | `status` | incremental state: uncompiled raw files, source-dir diff, affected wiki pages |
37
- | `export` | export the wiki graph as GraphML, Cypher, or a self-contained interactive HTML viewer |
38
91
  | `graph` | query `graph.json` with `path` / `neighbors` / `hubs` (zero-LLM traversal) |
39
- | `embed` | compute/update page embeddings (wiki/.vectors.json) for optional vector page location |
40
- | `mcp` | run a read-only MCP server (stdio) over the KB — for Cursor/Windsurf and other MCP-only agents |
41
-
42
- `ask` supports `-k <n>` (pages to load) and `--retrieve-only` (locate pages by
43
- BM25 without calling the LLM). All commands take `--kb <dir>` (default `.`).
44
-
45
- Retrieval is lexical (BM25). When it finds nothing — typically a question
46
- asked in a different language than the KB pages, or fully rephrased — `ask`
47
- falls back to letting the model pick pages from the KB listing
48
- (`llms.txt`), then answers from those pages as usual; only ids of real,
49
- non-invalidated pages are accepted. `--retrieve-only` stays pure BM25 while
50
- vector location is off (the default).
51
- Pages are always sent whole; when the selected pages exceed
52
- `askTokenBudget` (`wiki.config.json`, default 32000), the lowest-ranked
53
- pages are dropped, never truncated.
54
-
55
- **Optional vector page location** (v2.1): set `"vectorEnabled": true` in
56
- `wiki.config.json`, add `"embeddingModel"` to your provider in
57
- `~/.llm-wiki/config.json`, and run `npx @sdsrs/llm-wiki@0 embed`. `ask` then
58
- fuses BM25 with whole-page cosine similarity (RRF) — fixing cross-language and
59
- rephrased queries — and `--retrieve-only` labels each hit `[bm25]`, `[vector]`
60
- or `[bm25+vector]`. Vectors only locate pages; context is still whole pages,
92
+ | `embed` | compute/update page embeddings (`wiki/.vectors.json`) for optional vector location |
93
+ | `export` | export the graph as GraphML, Cypher, or an interactive HTML viewer; or the wiki as standard-markdown copies |
94
+ | `mcp` | run a read-only MCP server (stdio) over the KB |
95
+
96
+ All commands take `--kb <dir>` (default `.`). `ask` supports `-k <n>` (pages to
97
+ load) and `--retrieve-only` (locate pages without calling the LLM).
98
+
99
+ `scan` also warns when the source looks multi-domain (mixed-language files or
100
+ dispersed wiki tags) and suggests splitting — one KB per domain.
101
+
102
+ ### Asking questions
103
+
104
+ Retrieval is lexical (BM25) by default. When it finds nothing — typically a
105
+ question asked in a different language than the KB pages, or fully rephrased —
106
+ `ask` falls back to letting the model pick pages from the KB listing
107
+ (`llms.txt`); only ids of real, non-invalidated pages are accepted. Pages are
108
+ always sent whole; over `askTokenBudget` (`wiki.config.json`, default 32000)
109
+ the lowest-ranked pages are dropped, never truncated.
110
+
111
+ ### Vector page location (optional)
112
+
113
+ Fixes cross-language and rephrased queries. Three steps:
114
+
115
+ 1. Set `"vectorEnabled": true` in `wiki.config.json`.
116
+ 2. Add `"embeddingModel"` to your provider in `~/.llm-wiki/config.json`.
117
+ 3. Run `npx @sdsrs/llm-wiki@0 embed`.
118
+
119
+ `ask` (and the MCP `wiki_search` tool) then fuse BM25 with whole-page cosine
120
+ similarity (RRF); `--retrieve-only` labels each hit `[bm25]`, `[vector]` or
121
+ `[bm25+vector]`. Vectors only *locate* pages — context is still whole pages,
61
122
  never chunks. Any vector-path failure falls back to BM25 with a warning.
62
123
 
63
124
  ### Graph queries
64
125
 
65
126
  `graph.json` (rebuilt by `index`) is a link graph over the pages — wikilinks,
66
- `superseded_by`, and typed relations. Query it without any LLM call:
127
+ `superseded_by`, and typed relations:
67
128
 
68
129
  ```sh
69
- npx @sdsrs/llm-wiki@0 graph hubs --kb ./kb --top 5 # most-connected pages
130
+ npx @sdsrs/llm-wiki@0 graph hubs --kb ./kb --top 5 # most-connected pages
70
131
  npx @sdsrs/llm-wiki@0 graph path <from-id> <to-id> --kb ./kb # shortest link chain
71
132
  npx @sdsrs/llm-wiki@0 graph neighbors <id> -d 2 --kb ./kb # pages within N hops
72
133
  ```
73
134
 
74
- `hubs` ranks pages by degree (raw files excluded), e.g.:
75
-
76
- ```
77
- 31 concepts/llm-wiki (in 10 / out 21) LLM Wiki
78
- 31 entities/karpathy (in 18 / out 13) Andrej Karpathy
79
- 21 entities/obsidian (in 10 / out 11) Obsidian
80
- 16 concepts/rag (in 4 / out 12) RAG（检索增强生成）
81
- 13 concepts/ingest-query-lint (in 3 / out 10) Ingest / Query / Lint 三大操作
82
- ```
83
-
84
- The same three queries are exposed to MCP agents as `wiki_graph`
85
- (`op: path | neighbors | hubs`).
86
-
87
- **Typed relations.** Body `[[wikilinks]]` capture that two pages relate; when the
88
- *kind* of link matters, record it in the page frontmatter (edges merge into
89
- `graph.json` on `index`):
135
+ When the *kind* of link matters, record it in page frontmatter (edges merge
136
+ into `graph.json` on `index`):
90
137
 
91
138
  ```yaml
92
139
  relations:
@@ -95,124 +142,68 @@ relations:
95
142
  confidence: inferred # extracted | inferred | ambiguous
96
143
  ```
97
144
 
98
- `type` must be in the `relationTypes` vocabulary in `wiki.config.json` (extend it
99
- when the domain needs it). `confidence` records provenance: `extracted` =
100
- CLI-derived structure (`source` / `superseded_by` edges), `inferred` = agent
101
- judgment (the default, also carried by wikilinks), `ambiguous` = flagged for a
102
- human to resolve. `llm-wiki lint` validates that each relation target exists and
103
- its `type` is in the vocabulary, and counts relation targets as incoming links for
104
- orphan detection.
105
-
106
- ## Obsidian integration
107
-
108
- An llm_wiki KB is a plain folder of markdown with YAML frontmatter, so it opens
109
- directly as an Obsidian vault — "Open folder as vault", point it at `./kb`. Don't
110
- pre-create or edit `.obsidian/` — let Obsidian create and manage its own config
111
- folder; llm-wiki never writes it.
112
-
113
- What lights up out of the box:
114
-
115
- - **Graph view + backlinks** from the path-style `[[wikilinks]]` the pages already
116
- use (`[[entities/karpathy]]`) — they resolve across the `sources/`, `entities/`,
117
- `concepts/`, `comparisons/` subfolders and populate the graph and backlinks pane.
118
- - **Properties panel** from the page frontmatter (`type`, `title`, `tags`, `created`,
119
- `updated`, …). `tags` and `aliases` are YAML string lists, exactly Obsidian's
120
- expected shape (no `#` prefix inside the frontmatter list).
121
- - **Bases** (a core plugin) reads those properties into table/card views you can
122
- filter on `type`, `tags`, or `status` — e.g. a base over `type == "concept"` with a
123
- filter `status != "invalidated"` to hide retired pages.
124
- - **Link previews**: the frontmatter `description` doubles as Obsidian's hover-preview
125
- text for a page.
126
-
127
- Optional per-page `aliases` (a YAML list of alternative names) feed Obsidian's link
128
- autocomplete and alternate-name resolution; add them at page creation for topics with
129
- well-known synonyms, translations, or abbreviations.
130
-
131
- Typed `relations` (a list of objects — `to`/`type`/`confidence`) are valid YAML and
132
- round-trip fine, but a list-of-objects renders best in source mode rather than the
133
- Properties panel; they are primarily consumed by `llm-wiki graph` / the MCP server, not
134
- eyeballed in Obsidian.
135
-
136
- **Positioning.** Obsidian is for browsing and annotation; agents (via the wiki-* skills)
137
- remain the writers of `wiki/` pages. If you edit pages by hand in Obsidian, run
138
- `npx @sdsrs/llm-wiki@0 index --kb <kb>` afterwards to rebuild `index.md`, `graph.json`,
139
- and `llms.txt`. There is deliberately no bidirectional sync: concurrent writes to the
140
- same file (agent and editor at once) can corrupt content silently, which is why it stays
141
- out of scope. For tools that need standard markdown links instead of wikilinks, export a
142
- converted copy: `npx @sdsrs/llm-wiki@0 export --format markdown --kb <kb>`. The default
143
- output dir is `<kb>/wiki-md/` (inside the KB so its `../raw/...` links resolve against the
144
- real raw layer); if you use the KB itself as an Obsidian vault, add `wiki-md/` to Obsidian's
145
- "Excluded files" so the converted copies aren't indexed as duplicate notes — or export with
146
- `--out` outside the KB, at the cost of `raw/` links not resolving.
147
-
148
- **Verification checklist.** The conventions above are derived from Obsidian's docs, not
149
- tested against a specific Obsidian build. Run this once after opening the vault to confirm
150
- them on your Obsidian version:
151
-
152
- - [ ] Open `./kb` as a vault ("Open folder as vault").
153
- - [ ] Graph view shows clusters of typed pages linked by wikilinks.
154
- - [ ] Open a page — the backlinks panel is non-empty for a linked page.
155
- - [ ] Create a Base filtered to `type == "concept"`.
156
- - [ ] Add the filter `status != "invalidated"` and confirm invalidated pages drop out.
157
-
158
- ## LLM config
145
+ `lint` validates each relation target and type; `ambiguous` marks edges for a
146
+ human to resolve.
159
147
 
160
- `ask` needs an OpenAI-compatible endpoint. Configure `~/.llm-wiki/config.json`:
148
+ ### Skills (Claude Code / Codex)
161
149
 
162
- ```json
163
- {
164
- "priority": ["openai", "openrouter"],
165
- "providers": {
166
- "openai": { "baseURL": "https://api.openai.com/v1", "apiKeyEnv": "OPENAI_API_KEY", "model": "gpt-4o-mini" },
167
- "openrouter": { "baseURL": "https://openrouter.ai/api/v1", "apiKeyEnv": "OPENROUTER_API_KEY", "model": "anthropic/claude-sonnet-5" }
168
- }
169
- }
170
- ```
171
-
172
- The first provider whose `apiKeyEnv` env var is set wins. Export the matching key
173
- (`OPENAI_API_KEY` / `OPENROUTER_API_KEY`). Proxies are honored via the standard
174
- `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars.
175
-
176
- ## Skills and connecting
150
+ The bundled skills (`wiki-build`, `wiki-ingest`, `wiki-query`, `wiki-lint`,
151
+ `wiki-connect`, `wiki-distill`) let a coding agent build and maintain the KB —
152
+ the CLI never LLM-writes pages itself.
177
153
 
178
- The skills (`wiki-build`, `wiki-ingest`, `wiki-query`, `wiki-lint`, `wiki-connect`,
179
- `wiki-distill`) let a coding agent build and maintain the KB. Two ways to get them:
180
-
181
- **As a Claude Code plugin** (recommended — updates with the repo):
154
+ As a Claude Code plugin (recommended — updates with the repo):
182
155
 
183
156
  ```
184
157
  /plugin marketplace add sdsrss/llm_wiki
185
158
  /plugin install llm-wiki
186
159
  ```
187
160
 
188
- **Copied into a project** (works for Codex and other agents too):
161
+ Or copied into a project (works for Codex and other agents too):
189
162
 
190
163
  ```sh
191
- npx @sdsrs/llm-wiki install-skills # copy wiki-* skills into ./.claude
192
- npx @sdsrs/llm-wiki connect <projectDir> --kb <path> # register a KB into a project's CLAUDE.md
164
+ npx @sdsrs/llm-wiki@0 install-skills # copy wiki-* skills into ./.claude
165
+ npx @sdsrs/llm-wiki@0 connect <projectDir> --kb <path> # register the KB in a project's CLAUDE.md
166
+ ```
167
+
168
+ ### MCP server (Cursor, Windsurf, …)
169
+
170
+ ```json
171
+ { "mcpServers": { "my-kb": { "command": "npx", "args": ["-y", "@sdsrs/llm-wiki@0", "mcp", "--kb", "/path/to/my-kb"] } } }
193
172
  ```
194
173
 
195
- > **Migration note (pre-0.2.0 checkouts):** `connect` blocks written before the
196
- > npm rename say `npx llm-wiki ...` — on npm that bare name is an **unrelated
197
- > third-party package**. Re-run `connect` once per project to refresh the
198
- > sentinel block (it is rewritten in place, idempotently).
174
+ Five read-only tools: `wiki_overview`, `wiki_search` (page locator — ids +
175
+ descriptions, never full text), `wiki_read_page`, `wiki_ask` (needs LLM
176
+ config), `wiki_graph`. Page content is served as untrusted data with an
177
+ explicit notice.
178
+
179
+ ### Obsidian
180
+
181
+ A KB is a plain folder of markdown with YAML frontmatter — open it directly
182
+ with "Open folder as vault". Graph view and backlinks light up from the
183
+ path-style `[[wikilinks]]`, the Properties panel from the frontmatter, and
184
+ Bases can filter on `type` / `tags` / `status` (e.g. hide
185
+ `status == "invalidated"`). Obsidian is for browsing and annotation; agents
186
+ remain the writers. If you hand-edit pages, run
187
+ `npx @sdsrs/llm-wiki@0 index` afterwards to rebuild the derived files.
199
188
 
200
- ## MCP server
189
+ ### LLM config
201
190
 
202
- For agents that speak MCP but cannot run the bundled skills (Cursor, Windsurf, …):
191
+ `ask` needs an OpenAI-compatible endpoint. Configure `~/.llm-wiki/config.json`:
203
192
 
204
193
  ```json
205
- { "mcpServers": { "my-kb": { "command": "npx", "args": ["-y", "@sdsrs/llm-wiki@0", "mcp", "--kb", "/path/to/my-kb"] } } }
194
+ {
195
+ "priority": ["openai", "openrouter"],
196
+ "providers": {
197
+ "openai": { "baseURL": "https://api.openai.com/v1", "apiKeyEnv": "OPENAI_API_KEY", "model": "gpt-4o-mini" },
198
+ "openrouter": { "baseURL": "https://openrouter.ai/api/v1", "apiKeyEnv": "OPENROUTER_API_KEY", "model": "anthropic/claude-sonnet-5" }
199
+ }
200
+ }
206
201
  ```
207
202
 
208
- Five read-only tools: `wiki_overview` (index catalog), `wiki_search` (BM25 page
209
- locator — ids + descriptions, never full text), `wiki_read_page` (one whole page
210
- by id), `wiki_ask` (one-shot Q&A with citations; needs `~/.llm-wiki/config.json`,
211
- errors with guidance otherwise), `wiki_graph` (zero-LLM link-graph queries —
212
- `op: path | neighbors | hubs` over `graph.json`). Page content is served as untrusted data with an
213
- explicit notice, mirroring the skills' prompt-injection posture.
203
+ The first provider whose `apiKeyEnv` env var is set wins. Proxies are honored
204
+ via the standard `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` env vars.
214
205
 
215
- ## KB layout
206
+ ### KB layout
216
207
 
217
208
  ```
218
209
  my-kb/
@@ -224,10 +215,51 @@ my-kb/
224
215
  wiki.config.json thresholds, batch size, language
225
216
  ```
226
217
 
218
+ ## FAQ
219
+
220
+ **Do I need an API key?**
221
+ Only for `ask` (answering) and `embed` (optional vectors). Everything else —
222
+ scan, convert, index, lint, graph, export, the skills' bookkeeping — is
223
+ zero-LLM and offline.
224
+
225
+ **Am I locked in?**
226
+ No. A KB is plain markdown + YAML frontmatter in a folder. It opens as an
227
+ Obsidian vault as-is, and `export --format markdown` produces a copy with
228
+ standard links for tools that don't speak wikilinks.
229
+
230
+ **Can I edit wiki pages by hand?**
231
+ Yes, but run `index` afterwards to rebuild `index.md` / `graph.json` /
232
+ `llms.txt`. There is deliberately no editor⇄agent bidirectional sync:
233
+ concurrent writes to the same file can corrupt content silently.
234
+
235
+ **What happens to outdated knowledge?**
236
+ It's never deleted. Pages get `status: invalidated` (optionally
237
+ `superseded_by: <new-page>`) and drop out of `llms.txt` and retrieval, while
238
+ their history stays browsable.
239
+
240
+ **Can one KB hold several domains?**
241
+ One KB per domain works best. `scan` warns when a source directory looks
242
+ multi-domain (mixed languages, dispersed tags) and suggests splitting.
243
+
244
+ **Why do the docs write `@sdsrs/llm-wiki@0` everywhere?**
245
+ The bare `llm-wiki` npm name belongs to an unrelated package, and pinning the
246
+ major (`@0`) keeps `npx` runs reproducible. If a project's `CLAUDE.md` still
247
+ contains a pre-0.2.0 `npx llm-wiki ...` block, re-run `connect` once — the
248
+ block is rewritten in place, idempotently.
249
+
250
+ **Is page content trusted?**
251
+ No. Source documents and wiki pages are treated as untrusted data in every
252
+ model-facing prompt (ask, MCP, skills), with an explicit
253
+ never-follow-instructions notice — a prompt-injection guardrail, not an
254
+ afterthought.
255
+
227
256
  ## Iron rules
228
257
 
229
258
  - `raw/` is immutable — agents read it, never modify it.
230
259
  - Pages are full and self-contained; retrieval and `ask` load whole pages, never chunks.
231
260
  - Source documents are untrusted: page content is data, never instructions to follow.
232
- - Knowledge is never deleted: outdated pages are marked `status: invalidated`
233
- (with optional `superseded_by`) and drop out of `llms.txt` and `ask` retrieval.
261
+ - Knowledge is never deleted — invalidate (and supersede), don't erase.
262
+
263
+ ## License
264
+
265
+ MIT
package/bin/llm-wiki.mjs CHANGED
@@ -31,6 +31,10 @@ program.command('scan <srcDir>')
31
31
  console.log(`files: ${r.files.length} (skipped ${r.skipped.length})`)
32
32
  console.log(`duplicates: ${r.duplicates.exact.length} exact, ${r.duplicates.near.length} near`)
33
33
  console.log(`incremental: +${r.incremental.added} ~${r.incremental.changed} -${r.incremental.removed} =${r.incremental.unchanged}`)
34
+ const dm = r.domainMixture
35
+ if (dm.language.flagged) console.log(`warning: mixed-language source (zh ${dm.language.zh} / en ${dm.language.en} files) — this KB may span multiple domains`)
36
+ if (dm.tags?.flagged) console.log(`warning: wiki tags are dispersed (top tag covers ${Math.round(dm.tags.topShare * 100)}% of ${dm.tags.pages} pages) — this KB may span multiple domains`)
37
+ if (dm.flagged) console.log('hint: llm-wiki works best with one domain per KB; consider splitting the source into separate KBs')
34
38
  console.log(`compile plan: ${r.batches.length} batches -> ~${r.estimate.inputTokens} in / ~${r.estimate.outputTokens} out tokens`)
35
39
  console.log(`plan saved to ${opts.kb}/.scan-plan.json`)
36
40
  })
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@sdsrs/llm-wiki",
3
- "version": "0.6.0",
3
+ "version": "0.6.1",
4
4
  "description": "Compile messy document directories into Karpathy-style llm_wiki knowledge bases — standalone Q&A CLI + Claude Code/Codex skills",
5
5
  "directories": {
6
6
  "test": "test"
package/src/scanner.mjs CHANGED
@@ -5,9 +5,39 @@ import { loadKbConfig } from './templates.mjs'
5
5
  import { SUPPORTED_EXTS } from './convert.mjs'
6
6
  import { sha256File, minhashSignature, jaccardEstimate } from './hashing.mjs'
7
7
  import { loadManifest, diffManifest } from './manifest.mjs'
8
+ import { listWikiPages, isInvalidated } from './pages.mjs'
8
9
 
9
10
  const TEXT_EXTS = ['.md', '.markdown', '.txt', '.html', '.htm']
10
11
  const NEAR_DUP_THRESHOLD = 0.85
12
+ const LANG_MIX_MIN_FILES = 3
13
+ const LANG_MIX_MINORITY_SHARE = 0.25
14
+ const TAG_MIN_PAGES = 10
15
+ const TAG_TOP_SHARE_MIN = 0.3
16
+
17
+ // Advisory guardrail for the one-KB-per-domain rule, from signals scan already
18
+ // has: language mix across scanned text files, and tag dispersion across the
19
+ // built wiki (a cohesive KB has at least one tag most pages share). No LLM,
20
+ // no network; nothing blocks on it.
21
+ function detectDomainMixture(files, kbRoot) {
22
+ const zh = files.filter(f => f.lang === 'zh').length
23
+ const en = files.filter(f => f.lang === 'en').length
24
+ const minority = Math.min(zh, en)
25
+ const language = {
26
+ zh,
27
+ en,
28
+ flagged: minority >= LANG_MIX_MIN_FILES && minority / (zh + en) >= LANG_MIX_MINORITY_SHARE,
29
+ }
30
+ let tags = null
31
+ const pages = listWikiPages(kbRoot).filter(p =>
32
+ !p.error && !isInvalidated(p) && Array.isArray(p.data.tags) && p.data.tags.length > 0)
33
+ if (pages.length >= TAG_MIN_PAGES) {
34
+ const counts = new Map()
35
+ for (const p of pages) for (const t of new Set(p.data.tags)) counts.set(t, (counts.get(t) ?? 0) + 1)
36
+ const topShare = Math.max(...counts.values()) / pages.length
37
+ tags = { pages: pages.length, distinct: counts.size, topShare: Number(topShare.toFixed(2)), flagged: topShare < TAG_TOP_SHARE_MIN }
38
+ }
39
+ return { language, tags, flagged: language.flagged || (tags?.flagged ?? false) }
40
+ }
11
41
 
12
42
  export function estimateTokens(text) {
13
43
  let cjk = 0
@@ -103,6 +133,7 @@ export async function scanSource(srcDir, kbRoot, { exclude = [], persist = true
103
133
  skipped,
104
134
  duplicates: { exact, near },
105
135
  incremental: { added: diff.added.length, changed: diff.changed.length, removed: diff.removed.length, unchanged: diff.unchanged.length },
136
+ domainMixture: detectDomainMixture(files, kbRoot),
106
137
  batches,
107
138
  estimate,
108
139
  }