@sdsrs/llm-wiki 0.6.4 → 0.6.6

package/CHANGELOG.md

@@ -1,5 +1,71 @@
 # Changelog
 
+## 0.6.6 (2026-07-11)
+
+Audit remediation batch (roadmap M2 complete + M3/M4). No breaking changes, no KB
+migration; new config keys default to prior behavior. Suite 185 → 208.
+
+- **feat(cli): `llm-wiki --version`** prints the package version (was an unknown
+  option).
+- **fix(net): LLM and embedding calls now time out and retry.** `chat/completions`
+  and `embeddings` requests go through a shared helper with an `AbortSignal.timeout`
+  ceiling (120s) and exponential-backoff retry on 429 / 5xx / network errors — a
+  hung or rate-limited provider no longer hangs the CLI or aborts a whole `embed`
+  run on one transient failure.
+- **fix(embed): incremental durability.** `embed` now persists the vector store
+  after each batch, so a later batch failing keeps earlier batches' work (the
+  re-run reuses them by content hash — no repeated API cost). The store is written
+  atomically (temp + rename), as is `.manifest.json`.
+- **fix(manifest): tolerate a hand-edited manifest** missing or mistyping the
+  `files` key instead of throwing deep in the diff.
+- **fix(index): no line injection from frontmatter.** Newlines in a page's
+  title/description are collapsed before they are written into `index.md` /
+  `llms.txt`, so a crafted value can't inject headings or spoof wikilinks. Stale
+  `topics/*.md` are pruned when a type empties or the KB drops back below the split
+  threshold.
+- **fix(export): escape newlines in Cypher** node titles so they can't split a
+  statement.
+- **fix(docs): `@0` pin in the generated per-KB README**; corrected a fabricated
+  `supersedes` relation type in the main README to the real vocabulary (with a note
+  that `superseded_by` is a structural edge, not a `relations:` type); command
+  reference now lists `connect` / `install-skills` and documents `--version` /
+  `--follow-symlinks`; added an "Operational envelope" section (scale, no concurrent
+  writers, size cap, kana/hangul BM25 gap).
+- Test coverage: `installSkills` + CLI-layer smokes for `index` / `lint` / `status`
+  / `connect` / `install-skills` / `export` / `graph` and the `-k` / `--depth` /
+  `--top` guards; a timeout on the MCP stdio handshake test.
+
+## 0.6.5 (2026-07-11)
+
+Three security/robustness fixes from a full architecture-and-security audit
+(M1, the release-blocking milestone). Suite 185 → 187.
+
+**Behavior change (opt-out available):** `scan` now refuses a **symlinked source
+file whose target resolves outside the source directory**, instead of silently
+reading it into `raw/` and into a publishable wiki page. An attacker-supplied
+corpus containing e.g. `notes.md -> ~/.llm-wiki/config.json` (or `~/.ssh/id_rsa`,
+`/dev/zero`) could otherwise exfiltrate the victim's API key into a shared KB or
+hang the process. Escaping links now appear in the scan report as
+`skipped (symlink escapes source dir)`. **To restore the old follow-anywhere
+behavior for a trusted, curated corpus, pass `--follow-symlinks`.** In-tree
+symlinks are unaffected.
+
+- **fix(scan): symlinked files escaping the source tree are refused** (audit
+  HIGH-1). `walk()` blocked symlinked *directories* (loop safety) but followed
+  symlinked *files* and read the link target; the content was copied into `raw/`
+  and `raw/_originals/`. `scan` now `realpath`-resolves each symlinked file and
+  skips any whose target is outside the source dir; `--follow-symlinks` opts back
+  in.
+- **fix(scan): a file-size cap** (audit MEDIUM-1). Reads were unbounded — a
+  multi-GB (or `/dev/zero`-symlinked) file could OOM/hang the process. Files over
+  `maxFileBytes` (new `wiki.config.json` key, default 50 MB) are now skipped with
+  a clear reason rather than read whole.
+- **fix(export): `--out` managed-layer guard is now a prefix check** (audit
+  MEDIUM-2). The guard rejected `--out <kb>/raw` and `--out <kb>/wiki` exactly
+  but let a *subdirectory* (`--out <kb>/raw/sub`) through, where the export's
+  re-run `rmSync` would then wipe part of the immutable `raw/` layer. It now
+  rejects the KB root and anything inside `raw/`/`wiki/` at any depth.
+
 ## 0.6.4 (2026-07-11)
 
 Six fixes from a full-pipeline self-test pass (as a real user across every

package/README.md

@@ -65,8 +65,9 @@ npx @sdsrs/llm-wiki@0 ask "what did we decide about X?"
   (28 probes), enabling vectors lifted Recall@5 from 0.73 to 0.97 — the gap
   was almost entirely cross-language queries.
 - **A real link graph, queried without an LLM** — `graph path | neighbors |
-  hubs` traverse wikilinks and typed relations (`implements`, `supersedes`, …)
-  instantly.
+  hubs` traverse wikilinks and typed relations (`implements`, `contrasts_with`,
+  … — the `relationTypes` vocabulary; `superseded_by` is a separate structural
+  edge the CLI derives from frontmatter, not a `relations:` type) instantly.
 - **Agent-native, tool-agnostic** — the same KB serves a standalone CLI,
   Claude Code / Codex skills, an MCP server (Cursor, Windsurf, …), and opens
   directly as an Obsidian vault.
@@ -91,14 +92,31 @@ npx @sdsrs/llm-wiki@0 ask "what did we decide about X?"
 | `graph` | query `graph.json` with `path` / `neighbors` / `hubs` (zero-LLM traversal) |
 | `embed` | compute/update page embeddings (`wiki/.vectors.json`) for optional vector location |
 | `export` | export the graph as GraphML, Cypher, JSON Canvas, or an interactive HTML viewer; or the wiki as standard-markdown copies |
+| `connect <projectDir>` | register a KB into a project's `CLAUDE.md` (sentinel block) so coding agents use it |
+| `install-skills` | copy the bundled `wiki-*` skills into a `.claude` directory |
 | `mcp` | run a read-only MCP server (stdio) over the KB |
 
-All commands take `--kb <dir>` (default `.`). `ask` supports `-k <n>` (pages to
-load) and `--retrieve-only` (locate pages without calling the LLM).
+Run `llm-wiki --version` to print the version. All commands take `--kb <dir>`
+(default `.`). `ask` supports `-k <n>` (pages to load) and `--retrieve-only`
+(locate pages without calling the LLM). `scan` supports `--exclude <pattern...>`
+and `--follow-symlinks` (follow symlinked source files that resolve outside the
+source dir — off by default, since such a link can read an arbitrary file into
+the KB).
 
 `scan` also warns when the source looks multi-domain (mixed-language files or
 dispersed wiki tags) and suggests splitting — one KB per domain.
 
+### Operational envelope
+
+llm-wiki targets **single-user, single-domain KBs at the hundreds-of-pages
+scale**. Within that envelope everything is in-memory and re-read per operation
+(no persistent index); there is no locking, so **do not run two writers against
+one KB concurrently** (a CLI build alongside a long-lived MCP server is fine —
+MCP is read-only). Source files above `maxFileBytes` (`wiki.config.json`, default
+50 MB) are skipped. `wiki/log.md` is append-only and not rotated. BM25 tokenizes
+CJK by single char + bigram and ASCII by word, so purely **kana/hangul** queries
+won't match — enable vector location for robust cross-language retrieval.
+
 ### Asking questions
 
 Retrieval is lexical (BM25) by default. When it finds nothing — typically a

package/bin/llm-wiki.mjs

@@ -1,4 +1,5 @@
 #!/usr/bin/env node
+import fs from 'node:fs'
 import path from 'node:path'
 import { fileURLToPath } from 'node:url'
 import { program } from 'commander'
@@ -15,7 +16,9 @@ import { exportGraph, loadGraph, exportMarkdownPages } from '../src/export.mjs'
 import { shortestPath, neighborhood, hubs } from '../src/graph.mjs'
 import { runMcpServer } from '../src/mcp.mjs'
 
-program.name('llm-wiki').description('Compile messy directories into an llm_wiki knowledge base')
+const pkg = JSON.parse(fs.readFileSync(new URL('../package.json', import.meta.url), 'utf8'))
+
+program.name('llm-wiki').description('Compile messy directories into an llm_wiki knowledge base').version(pkg.version)
 
 program.command('init [dir]').description('scaffold a knowledge base').action((dir = '.') => {
   const { created, skipped } = initKb(dir)
@@ -26,8 +29,9 @@ program.command('scan <srcDir>')
   .description('inventory a source directory: dedup, batches, token estimate')
   .option('--kb <dir>', 'knowledge base root', '.')
   .option('--exclude <pattern...>', 'substring patterns to skip')
+  .option('--follow-symlinks', 'follow symlinked files that resolve outside the source dir (unsafe: only for trusted corpora)')
   .action(async (srcDir, opts) => {
-    const r = await scanSource(srcDir, opts.kb, { exclude: opts.exclude ?? [] })
+    const r = await scanSource(srcDir, opts.kb, { exclude: opts.exclude ?? [], followSymlinks: !!opts.followSymlinks })
     console.log(`files: ${r.files.length} (skipped ${r.skipped.length})`)
     console.log(`duplicates: ${r.duplicates.exact.length} exact, ${r.duplicates.near.length} near`)
     console.log(`incremental: +${r.incremental.added} ~${r.incremental.changed} -${r.incremental.removed} =${r.incremental.unchanged}`)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sdsrs/llm-wiki",
-  "version": "0.6.4",
+  "version": "0.6.6",
   "description": "Compile messy document directories into Karpathy-style llm_wiki knowledge bases — standalone Q&A CLI + Claude Code/Codex skills",
   "directories": {
     "test": "test"

package/src/ask.mjs

@@ -8,6 +8,7 @@ import { estimateTokens } from './scanner.mjs'
 import { loadLlmConfig, makeTransport } from './llm-config.mjs'
 import { loadVectorStore, normalize, cosineTopK } from './vector.mjs'
 import { embedTexts } from './embed.mjs'
+import { fetchWithRetry } from './retry.mjs'
 
 export function retrievePages(kbRoot, question, k = 6) {
   const pages = listWikiPages(kbRoot).filter(p => !p.error && !isInvalidated(p))
@@ -39,7 +40,7 @@ export function rrfFuse(lists, k) {
 // stderr warning. 'bm25': lexical only, returns before any vector access.
 // 'hybrid': explicit BM25+vector fusion — every missing prerequisite (and any
 // vector error) throws instead of degrading, and vectorEnabled is ignored.
-export async function locatePages(kbRoot, question, { k = 6, fetchImpl, retrieval = 'auto' } = {}) {
+export async function locatePages(kbRoot, question, { k = 6, fetchImpl, retrieval = 'auto', retry } = {}) {
   if (!['auto', 'bm25', 'hybrid'].includes(retrieval)) throw new Error(`unknown retrieval mode: ${retrieval}`)
   const bm25 = retrievePages(kbRoot, question, k)
   const asBm25 = () => ({ hits: bm25.map(h => ({ ...h, sources: ['bm25'] })), usedVector: false })
@@ -59,7 +60,7 @@ export async function locatePages(kbRoot, question, { k = 6, fetchImpl, retrieva
   // fusing it produces silent garbage, so treat it as missing (not a failure).
   if (store.model !== cfg.embeddingModel) return unavailable(`vector store was built with "${store.model}" but config says "${cfg.embeddingModel}" — re-run \`llm-wiki embed\``)
   try {
-    const t = fetchImpl ? { fetchImpl, dispatcher: undefined } : await makeTransport()
+    const t = fetchImpl ? { fetchImpl, dispatcher: undefined, retry } : { ...(await makeTransport()), retry }
     const [qv] = await embedTexts(cfg, t, [question])
     const qn = normalize(qv)
     const vecHits = qn ? cosineTopK(qn, store, k).map(v => ({ relPath: v.id, score: v.score })) : []
@@ -78,12 +79,11 @@ export async function locatePages(kbRoot, question, { k = 6, fetchImpl, retrieva
 }
 
 export async function chatCompletion(cfg, t, messages) {
-  const res = await t.fetchImpl(`${cfg.baseURL.replace(/\/$/, '')}/chat/completions`, {
+  const res = await fetchWithRetry(t.fetchImpl, `${cfg.baseURL.replace(/\/$/, '')}/chat/completions`, {
     method: 'POST',
     headers: { 'content-type': 'application/json', authorization: `Bearer ${cfg.apiKey}` },
     body: JSON.stringify({ model: cfg.model, messages }),
-    ...(t.dispatcher ? { dispatcher: t.dispatcher } : {}),
-  })
+  }, { dispatcher: t.dispatcher, ...(t.retry ?? {}) })
   if (!res.ok) {
     let body = ''
     try { body = (await res.text()).slice(0, 200) } catch { /* body unreadable; status alone */ }
@@ -130,9 +130,9 @@ async function pickPagesFromListing(p, question, k, cfg, t, validIds) {
   return picked.map(id => ({ relPath: `${id}.md`, score: 0 }))
 }
 
-export async function askKb(kbRoot, question, { k = 6, retrieveOnly = false, fetchImpl, retrieval = 'auto' } = {}) {
+export async function askKb(kbRoot, question, { k = 6, retrieveOnly = false, fetchImpl, retrieval = 'auto', retry } = {}) {
   const p = kbPaths(kbRoot)
-  let { hits } = await locatePages(kbRoot, question, { k, fetchImpl, retrieval })
+  let { hits } = await locatePages(kbRoot, question, { k, fetchImpl, retrieval, retry })
   if (retrieveOnly) return { pages: hits, answer: null }
   let validIds
   if (hits.length === 0) {
@@ -145,7 +145,7 @@ export async function askKb(kbRoot, question, { k = 6, retrieveOnly = false, fet
   if (!cfg) throw new Error('No LLM configured. Create ~/.llm-wiki/config.json with {"baseURL","apiKey","model"} (OpenAI-compatible).')
   // Injected fetchImpl (tests) is used as-is with no dispatcher; otherwise
   // pick the proxy-aware transport (undici fetch + agent, or global fetch).
-  const t = fetchImpl ? { fetchImpl, dispatcher: undefined } : await makeTransport()
+  const t = fetchImpl ? { fetchImpl, dispatcher: undefined, retry } : { ...(await makeTransport()), retry }
   let fallback = false
   if (hits.length === 0) {
     hits = await pickPagesFromListing(p, question, k, cfg, t, validIds)

package/src/embed.mjs

@@ -2,6 +2,7 @@ import { listWikiPages, isInvalidated } from './pages.mjs'
 import { loadLlmConfig, makeTransport } from './llm-config.mjs'
 import { sha256Text } from './hashing.mjs'
 import { normalize, pageEmbedText, loadVectorStore, saveVectorStore } from './vector.mjs'
+import { fetchWithRetry } from './retry.mjs'
 
 const BATCH = 64
 // Safety margin under the 8192-token input limit of common embedding models
@@ -44,12 +45,11 @@ function capEmbedText(text) {
 }
 
 export async function embedTexts(cfg, t, texts) {
-  const res = await t.fetchImpl(`${cfg.baseURL.replace(/\/$/, '')}/embeddings`, {
+  const res = await fetchWithRetry(t.fetchImpl, `${cfg.baseURL.replace(/\/$/, '')}/embeddings`, {
     method: 'POST',
     headers: { 'content-type': 'application/json', authorization: `Bearer ${cfg.apiKey}` },
     body: JSON.stringify({ model: cfg.embeddingModel, input: texts }),
-    ...(t.dispatcher ? { dispatcher: t.dispatcher } : {}),
-  })
+  }, { dispatcher: t.dispatcher, ...(t.retry ?? {}) })
   if (!res.ok) {
     let body = ''
     try { body = (await res.text()).slice(0, 200) } catch { /* status alone */ }
@@ -62,7 +62,7 @@ export async function embedTexts(cfg, t, texts) {
   return [...data.data].sort((a, b) => a.index - b.index).map(d => d.embedding)
 }
 
-export async function embedKb(kbRoot, { fetchImpl } = {}) {
+export async function embedKb(kbRoot, { fetchImpl, retry } = {}) {
   const cfg = loadLlmConfig(kbRoot)
   if (!cfg) throw new Error('No LLM configured. Create ~/.llm-wiki/config.json (OpenAI-compatible).')
   if (!cfg.embeddingModel) throw new Error('No embedding model configured. Add "embeddingModel" to your provider (or flat config) in ~/.llm-wiki/config.json, e.g. "text-embedding-3-small".')
@@ -84,7 +84,7 @@ export async function embedKb(kbRoot, { fetchImpl } = {}) {
   let dim = (prev && prev.model === cfg.embeddingModel) ? prev.dim : null
   let embedded = 0
   if (jobs.length > 0) {
-    const t = fetchImpl ? { fetchImpl, dispatcher: undefined } : await makeTransport()
+    const t = fetchImpl ? { fetchImpl, dispatcher: undefined, retry } : { ...(await makeTransport()), retry }
     for (let i = 0; i < jobs.length; i += BATCH) {
       const batch = jobs.slice(i, i + BATCH)
       const vecs = await embedTexts(cfg, t, batch.map(j => j.text))
@@ -96,6 +96,10 @@ export async function embedKb(kbRoot, { fetchImpl } = {}) {
         nextPages[j.relPath] = { hash: j.hash, vec: n }
         embedded++
       })
+      // Incremental durability: persist after each batch so a later batch failing
+      // (rate-limit, network) doesn't discard this batch's work — the re-run reuses
+      // it by hash (zero repeat API cost). saveVectorStore is atomic (temp+rename).
+      saveVectorStore(kbRoot, { model: cfg.embeddingModel, dim: dim ?? 0, pages: nextPages })
     }
   }
   const pruned = prev ? Object.keys(prev.pages).filter(id => !(id in nextPages)).length : 0

package/src/export.mjs

@@ -9,7 +9,7 @@ const xmlEscape = (s) => String(s)
   .replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;')
   .replace(/"/g, '&quot;').replace(/'/g, '&apos;')
 
-const cyEscape = (s) => String(s).replace(/\\/g, '\\\\').replace(/'/g, "\\'")
+const cyEscape = (s) => String(s).replace(/\\/g, '\\\\').replace(/\n/g, '\\n').replace(/\r/g, '\\r').replace(/'/g, "\\'")
 
 // graph.json only lists wiki pages as nodes, but source-type edges point at raw/
 // paths. Exports need every edge endpoint to exist, so synthesize raw nodes.
@@ -60,9 +60,16 @@ export function exportMarkdownPages(kbRoot, { out } = {}) {
   const p = kbPaths(kbRoot)
   const outDir = path.resolve(out ?? path.join(kbRoot, 'wiki-md'))
   // Never let --out resolve onto a managed KB layer: the marker guard below would
-  // later rmSync it, wiping the immutable raw/ inputs or the wiki/ pages themselves.
-  const forbidden = [path.resolve(kbRoot), path.resolve(kbRoot, 'raw'), path.resolve(p.wiki)]
-  if (forbidden.includes(outDir)) {
+  // later rmSync it, wiping the immutable raw/ inputs or the wiki/ pages. Prefix
+  // (not exact) match — `--out <kb>/raw/sub` is inside raw/ and just as destructive
+  // (audit MEDIUM-2). The KB root itself is rejected too.
+  const rawDir = path.resolve(kbRoot, 'raw')
+  const wikiDir = path.resolve(p.wiki)
+  const within = (parent, child) => {
+    const rel = path.relative(parent, child)
+    return rel === '' || (!rel.startsWith('..') && !path.isAbsolute(rel))
+  }
+  if (outDir === path.resolve(kbRoot) || within(rawDir, outDir) || within(wikiDir, outDir)) {
     throw new Error(`refusing to export into the KB's managed layers (${path.relative(kbRoot, outDir) || '.'}) — pass a dedicated --out directory`)
   }
   if (fs.existsSync(outDir) && !fs.statSync(outDir).isDirectory()) {

package/src/indexer.mjs

@@ -17,6 +17,11 @@ export function extractWikilinks(body) {
 
 const SECTION_TITLES = { source: 'Sources', entity: 'Entities', concept: 'Concepts', comparison: 'Comparisons', other: 'Other' }
 
+// Frontmatter title/description are interpolated raw into line-based markdown
+// (index.md, llms.txt). A value with newlines would inject extra lines, fake
+// headings or spoof wikilinks (audit LOW-1). Collapse any newline run to a space.
+const inline = (s) => String(s ?? '').replace(/[\r\n]+/g, ' ')
+
 export function buildIndex(kbRoot) {
   const p = kbPaths(kbRoot)
   const cfg = loadKbConfig(kbRoot)
@@ -42,7 +47,7 @@ export function buildIndex(kbRoot) {
   const byType = { source: [], entity: [], concept: [], comparison: [], other: [] }
   for (const pg of pages) (byType[pg.data.type] ?? byType.other).push(pg)
   const line = (pg) => {
-    const base = `- [[${pg.relPath.replace(/\.md$/, '')}]] — ${pg.data.description ?? ''}`
+    const base = `- [[${pg.relPath.replace(/\.md$/, '')}]] — ${inline(pg.data.description)}`
     if (!isInvalidated(pg)) return base
     return pg.data.superseded_by
       ? `${base} ⚠ invalidated, superseded by [[${pg.data.superseded_by}]]`
@@ -50,15 +55,27 @@ export function buildIndex(kbRoot) {
   }
 
   let indexBody = '# Index\n\n> Auto-generated by `llm-wiki index`. Only the "Pending concepts" section may be edited by the LLM.\n'
+  // Prune stale topics/*.md so a type that emptied (or a drop back below the split
+  // threshold) can't leave an orphaned list that exportMarkdownPages would copy.
+  const pruneTopics = (keep = new Set()) => {
+    if (!fs.existsSync(p.topics)) return
+    for (const f of fs.readdirSync(p.topics)) {
+      if (f.endsWith('.md') && !keep.has(f)) fs.rmSync(path.join(p.topics, f))
+    }
+  }
   const topicsSplit = pages.length > cfg.indexSplitAt
   if (topicsSplit) {
     fs.mkdirSync(p.topics, { recursive: true })
+    const written = new Set()
     for (const [type, list] of Object.entries(byType)) {
       if (!list.length) continue
       fs.writeFileSync(path.join(p.topics, `${type}.md`), `# ${SECTION_TITLES[type]}\n\n${list.map(line).join('\n')}\n`)
+      written.add(`${type}.md`)
       indexBody += `\n## ${SECTION_TITLES[type]}\nSee [[topics/${type}]] (${list.length} pages)\n`
     }
+    pruneTopics(written)
   } else {
+    pruneTopics() // no longer split: index.md inlines everything; drop any prior topic files
     for (const [type, list] of Object.entries(byType)) {
       if (type === 'other' && !list.length) continue
       indexBody += `\n## ${SECTION_TITLES[type]}\n${list.map(line).join('\n')}\n`
@@ -101,7 +118,7 @@ export function buildIndex(kbRoot) {
 
   const kbName = path.basename(path.resolve(kbRoot))
   const llms = [`# ${kbName}`, '', `> llm_wiki knowledge base. Read wiki/index.md first, then open full pages.`, '', '## Pages',
-    ...pages.filter(pg => !isInvalidated(pg)).map(pg => `- [${pg.data.title}](wiki/${pg.relPath}): ${pg.data.description ?? ''}`)].join('\n') + '\n'
+    ...pages.filter(pg => !isInvalidated(pg)).map(pg => `- [${inline(pg.data.title)}](wiki/${pg.relPath}): ${inline(pg.data.description)}`)].join('\n') + '\n'
   fs.writeFileSync(p.llmsTxt, llms)
 
   return { pageCount: pages.length, topicsSplit }

package/src/manifest.mjs

@@ -5,12 +5,19 @@ import { readJsonFile } from './json.mjs'
 export function loadManifest(kbRoot) {
   const p = kbPaths(kbRoot)
   if (!fs.existsSync(p.manifest)) return { files: {} }
-  return readJsonFile(p.manifest)
+  const m = readJsonFile(p.manifest)
+  // Shape guard: a hand-edited or half-written manifest lacking `files` (or with a
+  // non-object there) would make diffManifest's `Object.keys(manifest.files)` throw.
+  return (m && typeof m.files === 'object' && m.files !== null && !Array.isArray(m.files)) ? m : { files: {} }
 }
 
 export function saveManifest(kbRoot, manifest) {
   const p = kbPaths(kbRoot)
-  fs.writeFileSync(p.manifest, JSON.stringify(manifest, null, 2) + '\n')
+  // Atomic write: manifest is non-derived state (hash->raw map). A crash mid-write
+  // must not leave a truncated/corrupt file. Write a temp sibling, then rename.
+  const tmp = `${p.manifest}.tmp`
+  fs.writeFileSync(tmp, JSON.stringify(manifest, null, 2) + '\n')
+  fs.renameSync(tmp, p.manifest)
 }
 
 export function diffManifest(manifest, entries) {

package/src/mcp.mjs

@@ -17,7 +17,7 @@ const pkg = JSON.parse(fs.readFileSync(new URL('../package.json', import.meta.ur
 function textResult(text) { return { content: [{ type: 'text', text }] } }
 function errorResult(text) { return { content: [{ type: 'text', text }], isError: true } }
 
-export function createMcpServer(kbRoot, { fetchImpl } = {}) {
+export function createMcpServer(kbRoot, { fetchImpl, retry } = {}) {
   const p = kbPaths(kbRoot)
   const server = new McpServer({ name: 'llm-wiki', version: pkg.version })
 
@@ -38,7 +38,7 @@ export function createMcpServer(kbRoot, { fetchImpl } = {}) {
   }, async ({ query, k = 6 }) => {
     // 'auto' mode: opt-in via the KB's vectorEnabled, fail-open on any vector
     // error — KBs without embeddings keep the exact pre-v2.5 BM25 behavior.
-    const { hits, usedVector } = await locatePages(kbRoot, query, { k, ...(fetchImpl ? { fetchImpl } : {}) })
+    const { hits, usedVector } = await locatePages(kbRoot, query, { k, retry, ...(fetchImpl ? { fetchImpl } : {}) })
     if (hits.length === 0) {
       return textResult(usedVector
         ? 'No match from BM25 or vector retrieval. Call wiki_overview and pick pages from the catalog yourself.'
@@ -79,7 +79,7 @@ export function createMcpServer(kbRoot, { fetchImpl } = {}) {
     inputSchema: { question: z.string(), k: z.number().int().min(1).max(20).optional() },
   }, async ({ question, k = 6 }) => {
     try {
-      const r = await askKb(kbRoot, question, { k, ...(fetchImpl ? { fetchImpl } : {}) })
+      const r = await askKb(kbRoot, question, { k, retry, ...(fetchImpl ? { fetchImpl } : {}) })
       const parts = [DATA_NOTICE, '', r.answer, '', `--- pages used: ${r.pages.map(h => h.relPath).join(', ')}`]
       if (r.fallback) parts.push('(BM25 had no lexical match; pages were selected from the KB listing by the model)')
       if (r.trimmed?.length) parts.push(`(token budget: dropped ${r.trimmed.length} lower-ranked page(s): ${r.trimmed.join(', ')})`)

package/src/retry.mjs

@@ -0,0 +1,29 @@
+const RETRYABLE_STATUS = new Set([429, 500, 502, 503, 504])
+const sleep = (ms) => new Promise(r => setTimeout(r, ms))
+
+// One network call to an OpenAI-compatible endpoint, hardened for a CLI that must
+// not hang forever or die on a transient rate-limit. Per attempt: an
+// AbortSignal.timeout ceiling (its timer is unref'd, so it never keeps the process
+// alive); retry on 429/5xx and on network/abort errors with exponential backoff.
+// On a persistent 429/5xx the final response is returned so the caller raises its
+// own "API error <status>"; on a persistent network/abort error the last error is
+// thrown. retries=0 disables retry (immediate surface — used by tests).
+export async function fetchWithRetry(fetchImpl, url, opts, { dispatcher, timeoutMs = 120000, retries = 3, backoffMs = 500 } = {}) {
+  for (let attempt = 0; ; attempt++) {
+    try {
+      const res = await fetchImpl(url, {
+        ...opts,
+        signal: AbortSignal.timeout(timeoutMs),
+        ...(dispatcher ? { dispatcher } : {}),
+      })
+      if (RETRYABLE_STATUS.has(res.status) && attempt < retries) {
+        await sleep(backoffMs * 2 ** attempt)
+        continue
+      }
+      return res
+    } catch (err) {
+      if (attempt >= retries) throw err
+      await sleep(backoffMs * 2 ** attempt)
+    }
+  }
+}

package/src/scanner.mjs

@@ -51,38 +51,52 @@ export function estimateTokens(text) {
 
 // Symlinked directories are not followed (loop safety) but are recorded in
 // `skippedDirs` so they surface in the scan report instead of vanishing silently.
-// Symlinked files keep working: reads below follow the link as before.
-function* walk(dir, base = dir, skippedDirs = [], exclude = []) {
+// Symlinked files are followed ONLY when their target resolves inside the source
+// tree (`opts.rootReal`). A link escaping the tree would pull an arbitrary readable
+// file (e.g. ~/.llm-wiki/config.json, ~/.ssh/id_rsa, /dev/zero) into raw/ and into a
+// publishable page — the HIGH-1 exfiltration/DoS vector. `opts.followSymlinks` opts
+// back into follow-anywhere for a trusted, curated corpus.
+function* walk(dir, base, skippedDirs, exclude, opts) {
   for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
     if (e.name.startsWith('.') || e.name === 'node_modules') continue
     const abs = path.join(dir, e.name)
     if (e.isSymbolicLink()) {
       const rel = path.relative(base, abs)
       if (exclude.some(pat => rel.includes(pat))) { skippedDirs.push({ rel, reason: 'excluded' }); continue }
-      let st
-      try { st = fs.statSync(abs) } catch { skippedDirs.push({ rel, reason: 'broken symlink' }); continue }
+      let st, real
+      try { st = fs.statSync(abs); real = fs.realpathSync(abs) } catch { skippedDirs.push({ rel, reason: 'broken symlink' }); continue }
       if (st.isDirectory()) { skippedDirs.push({ rel, reason: 'symlinked directory (not followed)' }); continue }
+      if (!opts.followSymlinks) {
+        const relToRoot = path.relative(opts.rootReal, real)
+        if (relToRoot.startsWith('..') || path.isAbsolute(relToRoot)) {
+          skippedDirs.push({ rel, reason: 'symlink escapes source dir' }); continue
+        }
+      }
       yield rel
-    } else if (e.isDirectory()) yield* walk(abs, base, skippedDirs, exclude)
+    } else if (e.isDirectory()) yield* walk(abs, base, skippedDirs, exclude, opts)
     else yield path.relative(base, abs)
   }
 }
 
 // `persist: false` runs a read-only scan (e.g. for `status`) that does not
 // overwrite the .scan-plan.json a previous explicit `scan` produced.
-export async function scanSource(srcDir, kbRoot, { exclude = [], persist = true } = {}) {
+export async function scanSource(srcDir, kbRoot, { exclude = [], persist = true, followSymlinks = false } = {}) {
   const st = fs.statSync(srcDir, { throwIfNoEntry: false })
   if (!st) throw new Error(`source directory not found: ${srcDir}`)
   if (!st.isDirectory()) throw new Error(`source path is not a directory: ${srcDir}`)
   const cfg = loadKbConfig(kbRoot)
   const files = []
   const skipped = []
-  for (const rel of walk(srcDir, srcDir, skipped, exclude)) {
+  const rootReal = fs.realpathSync(srcDir)
+  for (const rel of walk(srcDir, srcDir, skipped, exclude, { followSymlinks, rootReal })) {
     if (exclude.some(pat => rel.includes(pat))) { skipped.push({ rel, reason: 'excluded' }); continue }
     const ext = path.extname(rel).toLowerCase()
     if (!SUPPORTED_EXTS.includes(ext)) { skipped.push({ rel, reason: `unsupported ${ext}` }); continue }
     const abs = path.join(srcDir, rel)
     const bytes = fs.statSync(abs).size
+    // Size cap (audit MEDIUM-1): an unbounded read OOMs on a hostile multi-GB file;
+    // gate before sha256File/readFileSync below, which both read the whole file.
+    if (bytes > cfg.maxFileBytes) { skipped.push({ rel, reason: `too large (${bytes} bytes > ${cfg.maxFileBytes} cap)` }); continue }
     const entry = { rel, ext, bytes, hash: sha256File(abs), lang: 'unknown', tokens: 0 }
     if (TEXT_EXTS.includes(ext)) {
       const text = fs.readFileSync(abs, 'utf8')

package/src/templates.mjs

@@ -11,6 +11,7 @@ export const DEFAULT_CONFIG = {
   language: 'auto',
   linkStyle: 'wikilink',
   askTokenBudget: 32000,
+  maxFileBytes: 52428800,
   vectorEnabled: false,
   relationTypes: ['implements', 'uses', 'depends_on', 'part_of', 'instance_of', 'derived_from', 'contrasts_with', 'causes'],
 }
@@ -117,7 +118,7 @@ export function readmeTemplate(name) {
 
 An llm_wiki knowledge base. Layers: \`raw/\` (immutable sources), \`wiki/\` (LLM-maintained pages).
 
-- Ask questions standalone: \`npx @sdsrs/llm-wiki ask "your question"\` (configure the LLM API in \`~/.llm-wiki/config.json\`).
+- Ask questions standalone: \`npx @sdsrs/llm-wiki@0 ask "your question"\` (configure the LLM API in \`~/.llm-wiki/config.json\`).
 - Browse: open \`wiki/index.md\`, or open this folder as an Obsidian vault.
 - Maintain with a coding agent: see \`AGENTS.md\`.
 `

package/src/vector.mjs

@@ -36,7 +36,13 @@ export function loadVectorStore(kbRoot) {
 export function saveVectorStore(kbRoot, store) {
   const rounded = { ...store, pages: Object.fromEntries(Object.entries(store.pages).map(([id, e]) =>
     [id, { hash: e.hash, vec: e.vec.map(x => Number(x.toFixed(5))) }])) }
-  fs.writeFileSync(vectorStorePath(kbRoot), JSON.stringify(rounded) + '\n')
+  // Atomic write: the per-batch flush in embedKb re-saves repeatedly; a crash
+  // mid-write must not leave a truncated store (loadVectorStore fails open, but
+  // temp+rename avoids the corruption entirely).
+  const f = vectorStorePath(kbRoot)
+  const tmp = `${f}.tmp`
+  fs.writeFileSync(tmp, JSON.stringify(rounded) + '\n')
+  fs.renameSync(tmp, f)
 }
 
 // queryVec must already be normalized; stored vecs are normalized at save time,