@sdsrs/llm-wiki 0.6.3 → 0.6.5

package/CHANGELOG.md

@@ -1,5 +1,68 @@
 # Changelog
 
+## 0.6.5 (2026-07-11)
+
+Three security/robustness fixes from a full architecture-and-security audit
+(M1, the release-blocking milestone). Suite 185 → 187.
+
+**Behavior change (opt-out available):** `scan` now refuses a **symlinked source
+file whose target resolves outside the source directory**, instead of silently
+reading it into `raw/` and into a publishable wiki page. An attacker-supplied
+corpus containing e.g. `notes.md -> ~/.llm-wiki/config.json` (or `~/.ssh/id_rsa`,
+`/dev/zero`) could otherwise exfiltrate the victim's API key into a shared KB or
+hang the process. Escaping links now appear in the scan report as
+`skipped (symlink escapes source dir)`. **To restore the old follow-anywhere
+behavior for a trusted, curated corpus, pass `--follow-symlinks`.** In-tree
+symlinks are unaffected.
+
+- **fix(scan): symlinked files escaping the source tree are refused** (audit
+  HIGH-1). `walk()` blocked symlinked *directories* (loop safety) but followed
+  symlinked *files* and read the link target; the content was copied into `raw/`
+  and `raw/_originals/`. `scan` now `realpath`-resolves each symlinked file and
+  skips any whose target is outside the source dir; `--follow-symlinks` opts back
+  in.
+- **fix(scan): a file-size cap** (audit MEDIUM-1). Reads were unbounded — a
+  multi-GB (or `/dev/zero`-symlinked) file could OOM/hang the process. Files over
+  `maxFileBytes` (new `wiki.config.json` key, default 50 MB) are now skipped with
+  a clear reason rather than read whole.
+- **fix(export): `--out` managed-layer guard is now a prefix check** (audit
+  MEDIUM-2). The guard rejected `--out <kb>/raw` and `--out <kb>/wiki` exactly
+  but let a *subdirectory* (`--out <kb>/raw/sub`) through, where the export's
+  re-run `rmSync` would then wipe part of the immutable `raw/` layer. It now
+  rejects the KB root and anything inside `raw/`/`wiki/` at any depth.
+
+## 0.6.4 (2026-07-11)
+
+Six fixes from a full-pipeline self-test pass (as a real user across every
+command). No breaking changes, no KB migration, no config change. Suite 179 →
+185.
+
+- **fix(convert): PDF conversion was 100% broken** and is now restored. The code
+  imported `pdf-parse/lib/pdf-parse.js` — the v1 subpath — but the pinned
+  dependency is pdf-parse v2, whose export map has no such subpath, so *every*
+  PDF failed at import time before any parsing. Switched to the v2 API
+  (`new PDFParse({ data }).getText()`). It slipped through because there was no
+  PDF fixture and the only test fed garbage bytes, which the import failure also
+  satisfied; added a real `test/fixtures/hello.pdf` + a positive extraction test.
+- fix(ask): `ask --retrieve-only` no longer prints nothing (exit 0) when
+  retrieval locates zero pages — the state right after `init`/`scan`, before the
+  wiki is built. It now emits a diagnostic to stderr (stdout stays pipe-clean),
+  matching the LLM path which already errored.
+- fix(convert): a source file whose whole basename is emoji/symbols
+  (e.g. `😀.md`) no longer produces a hidden `raw/.md` that `status` skips
+  (silently dropping the source from the incremental workflow). `slugify` now
+  falls back to `untitled`, so the existing `-N` collision handler yields
+  visible, shell-safe names (`untitled.md`, `untitled-2.md`, …).
+- fix(scan): a missing or non-directory source path now gives a human-readable
+  error ("source directory not found: X" / "source path is not a directory: X")
+  instead of a raw `ENOENT`/`ENOTDIR`; also improves `status --src`.
+- fix(export): `export --out path/with/missing/dirs/graph.graphml` now creates
+  the parent directory (matching the markdown-export sibling) instead of leaking
+  a raw `ENOENT`.
+- docs: the layout diagram placed `llms.txt` under `wiki/`, but `index` writes it
+  to the KB root (the llms.txt convention); corrected the diagram and command
+  table.
+
 ## 0.6.3 (2026-07-11)
 
 Two cosmetic fixes from an independent review of 0.6.0–0.6.2; no behavior

package/README.md

@@ -84,7 +84,7 @@ npx @sdsrs/llm-wiki@0 ask "what did we decide about X?"
 | `init [dir]` | scaffold a knowledge base (raw/, wiki/, AGENTS.md, wiki.config.json) |
 | `scan <srcDir>` | inventory a source dir: dedup, batches, incremental diff, token estimate |
 | `convert` | convert files from the scan plan into `raw/` markdown |
-| `index` | rebuild `wiki/index.md`, `graph.json`, `llms.txt` from page frontmatter |
+| `index` | rebuild `wiki/index.md`, `wiki/graph.json`, and root `llms.txt` from page frontmatter |
 | `ask <question>` | answer from the KB using full pages (never chunks), with citations |
 | `lint` | mechanical checks + a semantic worklist for the agent |
 | `status` | incremental state: uncompiled raw files, source-dir diff, affected wiki pages |
@@ -216,7 +216,8 @@ my-kb/
   raw/                 immutable converted source documents (agents only read)
   wiki/
     sources/  entities/  concepts/  comparisons/   full pages, one file each
-    index.md  graph.json  llms.txt                 generated by `index`
+    index.md  graph.json                           generated by `index`
+  llms.txt             flat page listing for agents, KB root (generated by `index`)
   AGENTS.md            the contract every maintaining agent follows
   wiki.config.json     thresholds, batch size, language
 ```

package/bin/llm-wiki.mjs

@@ -26,8 +26,9 @@ program.command('scan <srcDir>')
   .description('inventory a source directory: dedup, batches, token estimate')
   .option('--kb <dir>', 'knowledge base root', '.')
   .option('--exclude <pattern...>', 'substring patterns to skip')
+  .option('--follow-symlinks', 'follow symlinked files that resolve outside the source dir (unsafe: only for trusted corpora)')
   .action(async (srcDir, opts) => {
-    const r = await scanSource(srcDir, opts.kb, { exclude: opts.exclude ?? [] })
+    const r = await scanSource(srcDir, opts.kb, { exclude: opts.exclude ?? [], followSymlinks: !!opts.followSymlinks })
     console.log(`files: ${r.files.length} (skipped ${r.skipped.length})`)
     console.log(`duplicates: ${r.duplicates.exact.length} exact, ${r.duplicates.near.length} near`)
     console.log(`incremental: +${r.incremental.added} ~${r.incremental.changed} -${r.incremental.removed} =${r.incremental.unchanged}`)
@@ -74,6 +75,10 @@ program.command('ask <question>')
     if (!Number.isFinite(k) || k < 1) { console.error(`invalid -k value: ${opts.k} (expected a positive integer)`); process.exit(1) }
     const r = await askKb(opts.kb, question, { k, retrieveOnly: opts.retrieveOnly })
     if (opts.retrieveOnly) {
+      if (r.pages.length === 0) {
+        console.error('no pages located — the KB has no matching wiki pages (build them with the wiki-build skill, or check that wiki/ contains pages)')
+        return
+      }
       for (const h of r.pages) console.log(`${h.score.toFixed(3)}  ${h.relPath}  [${(h.sources ?? ['bm25']).join('+')}]`)
     } else {
       console.log(r.answer)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sdsrs/llm-wiki",
-  "version": "0.6.3",
+  "version": "0.6.5",
   "description": "Compile messy document directories into Karpathy-style llm_wiki knowledge bases — standalone Q&A CLI + Claude Code/Codex skills",
   "directories": {
     "test": "test"

package/src/convert.mjs

@@ -4,10 +4,14 @@ import path from 'node:path'
 export const SUPPORTED_EXTS = ['.md', '.markdown', '.txt', '.html', '.htm', '.pdf', '.docx']
 
 export function slugify(name) {
-  return name.toLowerCase()
+  const slug = name.toLowerCase()
     .replace(/\.[a-z0-9]+$/i, '')
     .replace(/[^\p{L}\p{N}]+/gu, '-')
     .replace(/^-+|-+$/g, '')
+  // A basename made entirely of emoji/symbols slugs to '' → a hidden `raw/.md`
+  // dotfile that `status` skips (source silently dropped). Never return empty;
+  // the caller's -N collision handler disambiguates untitled/untitled-2/…
+  return slug || 'untitled'
 }
 
 function titleFrom(markdown, fallback) {
@@ -44,9 +48,9 @@ export async function convertFile(srcPath) {
       return { markdown: `# ${article.title || base}\n\n${md}`, title: article.title || base, warnings }
     }
     if (ext === '.pdf') {
-      const pdfParse = (await import('pdf-parse/lib/pdf-parse.js')).default
-      const data = await pdfParse(fs.readFileSync(srcPath))
-      return { markdown: data.text, title: titleFrom(data.text, base), warnings }
+      const { PDFParse } = await import('pdf-parse')
+      const { text } = await new PDFParse({ data: fs.readFileSync(srcPath) }).getText()
+      return { markdown: text, title: titleFrom(text, base), warnings }
     }
     if (ext === '.docx') {
       const mammoth = await import('mammoth')

package/src/export.mjs

@@ -60,9 +60,16 @@ export function exportMarkdownPages(kbRoot, { out } = {}) {
   const p = kbPaths(kbRoot)
   const outDir = path.resolve(out ?? path.join(kbRoot, 'wiki-md'))
   // Never let --out resolve onto a managed KB layer: the marker guard below would
-  // later rmSync it, wiping the immutable raw/ inputs or the wiki/ pages themselves.
-  const forbidden = [path.resolve(kbRoot), path.resolve(kbRoot, 'raw'), path.resolve(p.wiki)]
-  if (forbidden.includes(outDir)) {
+  // later rmSync it, wiping the immutable raw/ inputs or the wiki/ pages. Prefix
+  // (not exact) match — `--out <kb>/raw/sub` is inside raw/ and just as destructive
+  // (audit MEDIUM-2). The KB root itself is rejected too.
+  const rawDir = path.resolve(kbRoot, 'raw')
+  const wikiDir = path.resolve(p.wiki)
+  const within = (parent, child) => {
+    const rel = path.relative(parent, child)
+    return rel === '' || (!rel.startsWith('..') && !path.isAbsolute(rel))
+  }
+  if (outDir === path.resolve(kbRoot) || within(rawDir, outDir) || within(wikiDir, outDir)) {
     throw new Error(`refusing to export into the KB's managed layers (${path.relative(kbRoot, outDir) || '.'}) — pass a dedicated --out directory`)
   }
   if (fs.existsSync(outDir) && !fs.statSync(outDir).isDirectory()) {
@@ -301,6 +308,9 @@ export function exportGraph(kbRoot, { format, out } = {}) {
   if (!render) throw new Error(`unknown format: ${format} (expected ${Object.keys(RENDERERS).join(' | ')} | markdown)`)
   const graph = loadGraph(kbRoot)
   const outPath = out ?? path.join(kbRoot, `graph.${EXT[format]}`)
+  // Match the markdown-export sibling: create the parent of an explicit --out so
+  // `--out new/dir/graph.graphml` writes instead of leaking a raw ENOENT.
+  fs.mkdirSync(path.dirname(path.resolve(outPath)), { recursive: true })
   fs.writeFileSync(outPath, render(graph))
   return { out: outPath, nodeCount: graph.nodes.length, edgeCount: graph.edges.length }
 }

package/src/scanner.mjs

@@ -51,35 +51,52 @@ export function estimateTokens(text) {
 
 // Symlinked directories are not followed (loop safety) but are recorded in
 // `skippedDirs` so they surface in the scan report instead of vanishing silently.
-// Symlinked files keep working: reads below follow the link as before.
-function* walk(dir, base = dir, skippedDirs = [], exclude = []) {
+// Symlinked files are followed ONLY when their target resolves inside the source
+// tree (`opts.rootReal`). A link escaping the tree would pull an arbitrary readable
+// file (e.g. ~/.llm-wiki/config.json, ~/.ssh/id_rsa, /dev/zero) into raw/ and into a
+// publishable page — the HIGH-1 exfiltration/DoS vector. `opts.followSymlinks` opts
+// back into follow-anywhere for a trusted, curated corpus.
+function* walk(dir, base, skippedDirs, exclude, opts) {
   for (const e of fs.readdirSync(dir, { withFileTypes: true })) {
     if (e.name.startsWith('.') || e.name === 'node_modules') continue
     const abs = path.join(dir, e.name)
     if (e.isSymbolicLink()) {
       const rel = path.relative(base, abs)
       if (exclude.some(pat => rel.includes(pat))) { skippedDirs.push({ rel, reason: 'excluded' }); continue }
-      let st
-      try { st = fs.statSync(abs) } catch { skippedDirs.push({ rel, reason: 'broken symlink' }); continue }
+      let st, real
+      try { st = fs.statSync(abs); real = fs.realpathSync(abs) } catch { skippedDirs.push({ rel, reason: 'broken symlink' }); continue }
       if (st.isDirectory()) { skippedDirs.push({ rel, reason: 'symlinked directory (not followed)' }); continue }
+      if (!opts.followSymlinks) {
+        const relToRoot = path.relative(opts.rootReal, real)
+        if (relToRoot.startsWith('..') || path.isAbsolute(relToRoot)) {
+          skippedDirs.push({ rel, reason: 'symlink escapes source dir' }); continue
+        }
+      }
       yield rel
-    } else if (e.isDirectory()) yield* walk(abs, base, skippedDirs, exclude)
+    } else if (e.isDirectory()) yield* walk(abs, base, skippedDirs, exclude, opts)
     else yield path.relative(base, abs)
   }
 }
 
 // `persist: false` runs a read-only scan (e.g. for `status`) that does not
 // overwrite the .scan-plan.json a previous explicit `scan` produced.
-export async function scanSource(srcDir, kbRoot, { exclude = [], persist = true } = {}) {
+export async function scanSource(srcDir, kbRoot, { exclude = [], persist = true, followSymlinks = false } = {}) {
+  const st = fs.statSync(srcDir, { throwIfNoEntry: false })
+  if (!st) throw new Error(`source directory not found: ${srcDir}`)
+  if (!st.isDirectory()) throw new Error(`source path is not a directory: ${srcDir}`)
   const cfg = loadKbConfig(kbRoot)
   const files = []
   const skipped = []
-  for (const rel of walk(srcDir, srcDir, skipped, exclude)) {
+  const rootReal = fs.realpathSync(srcDir)
+  for (const rel of walk(srcDir, srcDir, skipped, exclude, { followSymlinks, rootReal })) {
     if (exclude.some(pat => rel.includes(pat))) { skipped.push({ rel, reason: 'excluded' }); continue }
     const ext = path.extname(rel).toLowerCase()
     if (!SUPPORTED_EXTS.includes(ext)) { skipped.push({ rel, reason: `unsupported ${ext}` }); continue }
     const abs = path.join(srcDir, rel)
     const bytes = fs.statSync(abs).size
+    // Size cap (audit MEDIUM-1): an unbounded read OOMs on a hostile multi-GB file;
+    // gate before sha256File/readFileSync below, which both read the whole file.
+    if (bytes > cfg.maxFileBytes) { skipped.push({ rel, reason: `too large (${bytes} bytes > ${cfg.maxFileBytes} cap)` }); continue }
     const entry = { rel, ext, bytes, hash: sha256File(abs), lang: 'unknown', tokens: 0 }
     if (TEXT_EXTS.includes(ext)) {
       const text = fs.readFileSync(abs, 'utf8')

package/src/templates.mjs

@@ -11,6 +11,7 @@ export const DEFAULT_CONFIG = {
   language: 'auto',
   linkStyle: 'wikilink',
   askTokenBudget: 32000,
+  maxFileBytes: 52428800,
   vectorEnabled: false,
   relationTypes: ['implements', 'uses', 'depends_on', 'part_of', 'instance_of', 'derived_from', 'contrasts_with', 'causes'],
 }