pi-gauntlet 5.22.0 → 5.23.0

package/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## v5.23.0 - 2026-09-27
+
+### Added
+
+- `gauntlet-spec-index` gains `--corpus specs|docs` (default `specs`, output unchanged). `--corpus docs` indexes project documentation - tracked and untracked non-gitignored `*.md` files matching `**/doc/**/*.md`, `**/docs/**/*.md`, `README.md`, `AGENTS.md`, replaceable through `- docs: <glob>` bullets under `## Spec index` in the gauntlet overrides file - into a second FTS5 table in the same cache, minus `doc(s)/specs`, `doc(s)/plans`, `node_modules`, `.pi/gauntlet`, `.worktrees`, symlinks and context drafts, and prints `score`, `path`, `title`, `snippet`. Both corpora share one confidence rule driven by a corpus descriptor (strong fields `title`/`goal` for specs, `title`/`headings` for docs). The brainstorming scout runs the docs query and renders `Docs touched: <path> - <section heading>` lines that round 2 weighs as documentation-impact candidates.
+
+### Changed
+
+- Spec-index cache schema is 3 (the `files` table is keyed by `(corpus, path)`); existing caches rebuild on first query. `#`/`##`/`###` lines inside fenced code blocks no longer count as the title or headings in either corpus.
+
 ## v5.22.0 - 2026-09-26
 
 ### Added

package/README.md

@@ -137,7 +137,7 @@ Pin an exact release with `npm:pi-gauntlet@X.Y.Z`. See [doc/install-internals.md
 
 ## Spec search index
 
-`gauntlet-spec-index` provides lexical search across `doc/specs/*.md` at the repository root and one service level down. From a repository worktree, run `node <pi-gauntlet-package>/bin/gauntlet-spec-index.mjs --query "<text>" [--limit N] [--exclude <repo-relative path>]...`; it requires Node >=24.15.0, refreshes its FTS5 index on every query, and prints tab-separated `score`, `path`, `service`, `title`, `status`, `shipped_at`, `state`, `files`, and `snippet` columns. Rows pass a confidence rule before `--limit` applies: a query term is evidence when it occurs in fewer than half the indexed specs (stem variants of one word count once); query tokens split on word boundaries, so `gauntlet-spec-index` searches as three words; a row is returned only when it matches at least two distinct evidence terms in its `title` or `goal`, and rows scoring below half the best live row's score are dropped. A query with no real match therefore prints the header only and exits 0, and a corpus of two or fewer specs never returns rows. `--exclude` (repeatable) removes a path before the rule runs, so the spec under review never sets the bar. `state` is `superseded` when a `> **Superseded by:** ... - fully` banner sits in the block directly under the title and `live` otherwise; named-section banners and consumer-defined banner syntaxes both read as `live`. Live rows sort before superseded rows, then by score. The `files` column is a `;`-separated list of repo-relative paths the spec's shipped change modified and that still exist in the repository, the literal `missing` when the spec's telemetry record has no `derived.modified_files` list, or blank when there is no readable record or no recorded path remains. The per-worktree cache lives at `.pi/gauntlet/index.sqlite`, and its first creation adds `/.pi/gauntlet/index.sqlite*` to Git's `info/exclude` so the database and SQLite sidecars stay out of `git status`. `/skill:brainstorming` queries the index twice: the scout at gather time from the request, and the main loop at spec-writing from the finished spec's title, goal, and headings, surfacing new `live` candidates at the review gate.
+`gauntlet-spec-index` provides lexical search over two corpora in one per-worktree cache. From a repository worktree, run `node <pi-gauntlet-package>/bin/gauntlet-spec-index.mjs --query "<text>" [--corpus specs|docs] [--limit N] [--exclude <repo-relative path>]...`; it requires Node >=24.15.0 and refreshes its FTS5 index on every query. `--corpus specs` (the default) searches `doc/specs/*.md` at the repository root and one service level down and prints tab-separated `score`, `path`, `service`, `title`, `status`, `shipped_at`, `state`, `files`, and `snippet` columns. `--corpus docs` searches project documentation and prints `score`, `path`, `title`, and `snippet`; the snippet is the matched `##`/`###` heading line when one holds a query term, else a body fragment. Docs candidates are the tracked and untracked, non-gitignored `*.md` files (`git ls-files -co --exclude-standard`, so nested repositories and submodules are not indexed) matching an include list - default `**/doc/**/*.md`, `**/docs/**/*.md`, `README.md`, `AGENTS.md`, where a slash-free entry also matches one service level down - minus the fixed exclusions `**/doc/specs/**`, `**/docs/specs/**`, `**/doc/plans/**`, `**/docs/plans/**`, `**/node_modules/**`, `.pi/gauntlet/**`, `.worktrees/**`, symlinks, and files whose first line is the context-draft marker; `**` never matches a dot directory, so `.pi/*.md` or `.github/*.md` is indexed only through an explicit include. A project replaces the include list with `- docs: <glob>` bullets under a `## Spec index` heading in its gauntlet overrides file (see [Project-specific overrides](#project-specific-overrides)); the exclusions are not configurable. Both corpora pass the same confidence rule before `--limit` applies: a query term is evidence when it occurs in fewer than half the rows of the queried table (stem variants of one word count once); query tokens split on word boundaries, so `gauntlet-spec-index` searches as three words; a row is returned only when it matches at least two distinct evidence terms in the corpus's strong fields - `title` or `goal` for specs, `title` or `headings` for docs - and rows scoring below half the best live row's score are dropped (every docs row is live). A query with no real match therefore prints the header only and exits 0, and a corpus of two or fewer files never returns rows. `--exclude` (repeatable) removes a path before the rule runs, so the spec under review never sets the bar. `state` is `superseded` when a `> **Superseded by:** ... - fully` banner sits in the block directly under the title and `live` otherwise; named-section banners and consumer-defined banner syntaxes both read as `live`. Live rows sort before superseded rows, then by score. The `files` column is a `;`-separated list of repo-relative paths the spec's shipped change modified and that still exist in the repository, the literal `missing` when the spec's telemetry record has no `derived.modified_files` list, or blank when there is no readable record or no recorded path remains. The cache lives at `.pi/gauntlet/index.sqlite`, and its first creation adds `/.pi/gauntlet/index.sqlite*` to Git's `info/exclude` so the database and SQLite sidecars stay out of `git status`. `/skill:brainstorming` queries the specs corpus twice - the scout at gather time from the request, and the main loop at spec-writing from the finished spec's title, goal, and headings, surfacing new `live` candidates at the review gate - and the scout queries the docs corpus once to render `Docs touched:` lines that round 2 weighs as documentation-impact candidates.
 
 ## Performance digest
 
@@ -279,10 +279,18 @@ Use the project's wrapper: `script/worktree create <name>`. It provisions an iso
 database and copies `.env.local`. Never call `git worktree add` directly.
 ```
 
-Beyond `## conventions` and skill-named sections, headings a skill reads by name are documented in their owning skills; other headings retain topic/workflow-convention matching. The override file is read by skill instructions, not by the Pi runtime itself. Missing or empty `## conventions` adds no rules; a lower-priority file cannot supplement the selected file.
+Beyond `## conventions` and skill-named sections, headings a skill reads by name are documented in their owning skills; other headings retain topic/workflow-convention matching. The override file is read by skill instructions and, for its `## Spec index` section only, by `gauntlet-spec-index`; the Pi runtime itself never reads it. Missing or empty `## conventions` adds no rules; a lower-priority file cannot supplement the selected file.
 
 **Discovery ladder:** skills check three locations, in order, and use the first one found - never merged: `.pi/gauntlet-overrides.md`, then `<repo root>/gauntlet-overrides.md`, then `<repo root>/doc/gauntlet-overrides.md` (`<repo root>` = `git rev-parse --show-toplevel`, or the current directory outside a repo). Pick one location per repo.
 
+**`## Spec index` section:** `gauntlet-spec-index --corpus docs` reads `- docs: <glob>` bullets under this heading as the docs include list, replacing the default (`**/doc/**/*.md`, `**/docs/**/*.md`, `README.md`, `AGENTS.md`). Backticks or quotes around the glob are stripped; no other key under the heading is read, and the fixed exclusions always apply. For example:
+
+```markdown
+## Spec index
+- docs: `**/doc/**/*.md`
+- docs: `handbook/**/*.md`
+```
+
 **`## Issue tracker` section:** `shape-ticket` resolves tracker access through a capability ladder, and this is its first rung - it overrides the zero-config `gh` (GitHub) / `linearis` (Linear) defaults for any other tracker. Name the CLI's read, search, create, update, and post comment commands explicitly. For a Jira CLI, for example:
 
 ```markdown

package/bin/gauntlet-spec-index.mjs

@@ -1,13 +1,12 @@
 #!/usr/bin/env node
-// Lexical search over the spec corpus. Build-on-query: refresh a per-worktree
-// FTS5 cache by mtime+size, then rank with bm25 and join telemetry at output.
-import { readFileSync, appendFileSync, existsSync, statSync, readdirSync, mkdirSync, rmSync, realpathSync } from "node:fs";
-import { join, dirname, basename, isAbsolute } from "node:path";
+// Lexical search over spec and docs corpora (FTS5, bm25, shared confidence rule); telemetry joined for specs only.
+import { readFileSync, appendFileSync, existsSync, statSync, lstatSync, readdirSync, mkdirSync, rmSync, realpathSync, openSync, readSync, closeSync } from "node:fs";
+import { join, dirname, basename, isAbsolute, matchesGlob } from "node:path";
 import { execFileSync } from "node:child_process";
 import process from "node:process";
 import { parse as parseYaml } from "yaml";
 
-const SCHEMA_VERSION = 2;
+const SCHEMA_VERSION = 3;
 const EVIDENCE_DF_FRACTION = 0.5;
 const MIN_EVIDENCE_TOKENS = 2;
 const SCORE_RATIO = 0.5;
@@ -18,17 +17,23 @@ const MIN_NODE = [24, 15, 0];
 const DRAFT_MARKER = "# CONTEXT DRAFT - NOT A SPEC - fully replaced at spec-writing";
 const EXCLUDE_LINE = "/.pi/gauntlet/index.sqlite*";
 const SKIP_DIRS = new Set([".worktrees", "node_modules", "build"]);
-const HEADER = ["score", "path", "service", "title", "status", "shipped_at", "state", "files", "snippet"];
+const FENCE = /^\s*(`{3,}|~{3,})/;
+const DOC_INCLUDES = ["**/doc/**/*.md", "**/docs/**/*.md", "README.md", "AGENTS.md"];
+const DOC_EXCLUDES = ["**/doc/specs/**", "**/docs/specs/**", "**/doc/plans/**", "**/docs/plans/**", "**/node_modules/**", ".pi/gauntlet/**", ".worktrees/**"];
+const OVERRIDES_FILES = [".pi/gauntlet-overrides.md", "gauntlet-overrides.md", "doc/gauntlet-overrides.md"];
 const SCHEMA = `
   CREATE TABLE IF NOT EXISTS meta  (key TEXT PRIMARY KEY, value TEXT);
-  CREATE TABLE IF NOT EXISTS files (path TEXT PRIMARY KEY, mtime_ms INTEGER, size INTEGER);
+  CREATE TABLE IF NOT EXISTS files (corpus TEXT NOT NULL, path TEXT NOT NULL, mtime_ms INTEGER, size INTEGER, PRIMARY KEY (corpus, path));
   CREATE VIRTUAL TABLE IF NOT EXISTS specs USING fts5(
     path UNINDEXED, service UNINDEXED, title, goal, headings, body, state UNINDEXED,
     tokenize = 'porter unicode61');
+  CREATE VIRTUAL TABLE IF NOT EXISTS docs USING fts5(
+    path UNINDEXED, title, headings, body,
+    tokenize = 'porter unicode61');
 `;
 
 const usage = () => {
-  process.stderr.write('usage: gauntlet-spec-index --query "<text>" [--limit N] [--exclude <repo-relative path>]...\n');
+  process.stderr.write('usage: gauntlet-spec-index --query "<text>" [--corpus specs|docs] [--limit N] [--exclude <repo-relative path>]...\n');
   process.exit(1);
 };
 const die = (msg) => {
@@ -38,16 +43,18 @@ const die = (msg) => {
 
 function parseArgs(argv) {
   let query;
+  let corpus = "specs";
   let limit = 10;
   const exclude = [];
   for (let i = 0; i < argv.length; i++) {
     if (argv[i] === "--query" && argv[i + 1] !== undefined) query = argv[++i];
+    else if (argv[i] === "--corpus" && Object.hasOwn(CORPORA, argv[i + 1] ?? "")) corpus = argv[++i];
     else if (argv[i] === "--limit" && /^[1-9]\d*$/.test(argv[i + 1] ?? "")) limit = Number(argv[++i]);
     else if (argv[i] === "--exclude" && argv[i + 1] !== undefined) exclude.push(argv[++i]);
     else usage();
   }
   if (query === undefined) usage();
-  return { query, limit, exclude };
+  return { query, corpus, limit, exclude };
 }
 
 function nodeOk() {
@@ -88,11 +95,64 @@ function discover(root) {
     }
   };
   collect("doc/specs", "root");
-  const entries = readdirSync(root, { withFileTypes: true }).sort((a, b) => a.name.localeCompare(b.name));
-  for (const e of entries) {
-    if (e.name.startsWith(".") || SKIP_DIRS.has(e.name)) continue;
-    if (!e.isDirectory() && !e.isSymbolicLink()) continue;
-    collect(`${e.name}/doc/specs`, e.name);
+  for (const s of services(root).sort((a, b) => a.localeCompare(b))) collect(`${s}/doc/specs`, s);
+  return out;
+}
+
+// Top-level directories discover() would scan; a slash-free include also matches one level under them.
+function services(root) {
+  return readdirSync(root, { withFileTypes: true })
+    .filter((e) => (e.isDirectory() || e.isSymbolicLink()) && !e.name.startsWith(".") && !SKIP_DIRS.has(e.name))
+    .map((e) => e.name);
+}
+
+function includes(root) {
+  const file = OVERRIDES_FILES.map((rel) => join(root, rel)).find((abs) => existsSync(abs));
+  if (!file) return DOC_INCLUDES;
+  const globs = [];
+  let inSection = false;
+  let text;
+  try {
+    text = readFileSync(file, "utf8");
+  } catch (e) {
+    die(`gauntlet-spec-index: unreadable overrides file ${file}: ${e.message}`);
+  }
+  for (const line of text.split(/\r?\n/)) {
+    if (/^## /.test(line)) { inSection = line.trim() === "## Spec index"; continue; }
+    if (!inSection) continue;
+    const m = /^-\s+docs:\s*(.+?)\s*$/.exec(line);
+    if (m) globs.push(m[1].replace(/^[`"']+|[`"']+$/g, ""));
+  }
+  return globs.length ? globs : DOC_INCLUDES;
+}
+
+function firstLine(abs) {
+  const fd = openSync(abs, "r");
+  try {
+    const buf = Buffer.alloc(128);
+    const n = readSync(fd, buf, 0, 128, 0);
+    return buf.toString("utf8", 0, n).split(/\r?\n/, 1)[0];
+  } finally {
+    closeSync(fd);
+  }
+}
+
+function discoverDocs(root, globs) {
+  const svc = services(root);
+  const patterns = globs.flatMap((g) => (g.includes("/") ? [g] : [g, ...svc.map((s) => `${s}/${g}`)]));
+  const listed = execFileSync("git", ["-C", root, "ls-files", "-co", "--exclude-standard", "-z", "--", "*.md"], { encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).split("\0").filter(Boolean);
+  const out = [];
+  for (const rel of listed) {
+    if (!patterns.some((p) => matchesGlob(rel, p))) continue;
+    if (DOC_EXCLUDES.some((p) => matchesGlob(rel, p))) continue;
+    let st;
+    try {
+      st = lstatSync(join(root, rel));
+      if (!st.isFile() || firstLine(join(root, rel)) === DRAFT_MARKER) continue;
+    } catch {
+      continue;
+    }
+    out.push({ path: rel, mtime_ms: Math.trunc(st.mtimeMs), size: st.size });
   }
   return out;
 }
@@ -145,17 +205,34 @@ async function openDb(root) {
   }
 }
 
+function fenceMask(lines) {
+  const mask = new Array(lines.length).fill(false);
+  let open = null;
+  for (let i = 0; i < lines.length; i++) {
+    const m = FENCE.exec(lines[i]);
+    if (open) {
+      mask[i] = true;
+      if (m && m[1][0] === open.ch && m[1].length >= open.len) open = null;
+    } else if (m) {
+      mask[i] = true;
+      open = { ch: m[1][0], len: m[1].length };
+    }
+  }
+  return mask;
+}
+
 function extract(text, path) {
   const lines = text.split(/\r?\n/);
-  const h1 = lines.findIndex((l) => l.startsWith("# "));
+  const fenced = fenceMask(lines);
+  const h1 = lines.findIndex((l, i) => !fenced[i] && l.startsWith("# "));
   const title = (h1 >= 0 ? lines[h1].slice(2).trim() : "") || basename(path, ".md");
   const goal = lines.find((l) => l.startsWith("**Goal:**"))?.slice("**Goal:**".length).trim() ?? "";
-  const headings = lines.filter((l) => /^##{1,2} /.test(l)).map((l) => l.replace(/^#+ /, "")).join("\n");
+  const headings = lines.filter((l, i) => !fenced[i] && /^##{1,2} /.test(l)).map((l) => l.replace(/^#+ /, "")).join("\n");
   let state = "live";
   if (h1 >= 0) {
     // Title block: everything after the H1 up to the first H2 heading or code fence.
     for (const l of lines.slice(h1 + 1)) {
-      if (/^## /.test(l) || /^\s*(`{3,}|~{3,})/.test(l)) break;
+      if (/^## /.test(l) || FENCE.test(l)) break;
       if (FULLY_BANNER.test(l)) { state = "superseded"; break; }
     }
   }
@@ -163,24 +240,25 @@ function extract(text, path) {
 }
 
 function refresh(db, root, corpus) {
-  const known = new Map(db.prepare("SELECT path, mtime_ms, size FROM files").all().map((r) => [r.path, r]));
-  const present = new Set(corpus.map((f) => f.path));
-  const delSpec = db.prepare("DELETE FROM specs WHERE path = ?");
-  const delFile = db.prepare("DELETE FROM files WHERE path = ?");
-  const insSpec = db.prepare("INSERT INTO specs (path, service, title, goal, headings, body, state) VALUES (?, ?, ?, ?, ?, ?, ?)");
-  const putFile = db.prepare("INSERT OR REPLACE INTO files (path, mtime_ms, size) VALUES (?, ?, ?)");
+  const { table } = corpus;
+  const found = corpus.discover(root);
+  const known = new Map(db.prepare("SELECT path, mtime_ms, size FROM files WHERE corpus = ?").all(table).map((r) => [r.path, r]));
+  const present = new Set(found.map((f) => f.path));
+  const delRow = db.prepare(`DELETE FROM ${table} WHERE path = ?`);
+  const delFile = db.prepare("DELETE FROM files WHERE corpus = ? AND path = ?");
+  const insRow = db.prepare(`INSERT INTO ${table} (${corpus.columns.join(", ")}) VALUES (${corpus.columns.map(() => "?").join(", ")})`);
+  const putFile = db.prepare("INSERT OR REPLACE INTO files (corpus, path, mtime_ms, size) VALUES (?, ?, ?, ?)");
   db.exec("BEGIN IMMEDIATE");
   try {
-    for (const path of known.keys()) if (!present.has(path)) { delSpec.run(path); delFile.run(path); }
-    for (const f of corpus) {
+    for (const path of known.keys()) if (!present.has(path)) { delRow.run(path); delFile.run(table, path); }
+    for (const f of found) {
       const k = known.get(f.path);
       if (k && k.mtime_ms === f.mtime_ms && k.size === f.size) continue;
       const text = readFileSync(join(root, f.path), "utf8");
-      delSpec.run(f.path);
-      if (text.split(/\r?\n/, 1)[0] === DRAFT_MARKER) { delFile.run(f.path); continue; }
-      const x = extract(text, f.path);
-      insSpec.run(f.path, f.service, x.title, x.goal, x.headings, text, x.state);
-      putFile.run(f.path, f.mtime_ms, f.size);
+      delRow.run(f.path);
+      if (text.split(/\r?\n/, 1)[0] === DRAFT_MARKER) { delFile.run(table, f.path); continue; }
+      insRow.run(...corpus.row(text, f));
+      putFile.run(table, f.path, f.mtime_ms, f.size);
     }
     db.exec("COMMIT");
   } catch (e) {
@@ -212,32 +290,32 @@ function representatives(db, toks) {
 function confidenceFilter(rows, evidence) {
   return rows.filter((r) => {
     const hits = evidence.filter((e) => e.all.has(r.rowid)).length;
-    const titleGoal = evidence.filter((e) => e.titleGoal.has(r.rowid)).length;
-    return hits >= MIN_EVIDENCE_TOKENS && titleGoal >= MIN_EVIDENCE_TOKENS;
+    const strong = evidence.filter((e) => e.strong.has(r.rowid)).length;
+    return hits >= MIN_EVIDENCE_TOKENS && strong >= MIN_EVIDENCE_TOKENS;
   });
 }
 
-function query(db, toks, { limit, exclude }) {
-  const n = db.prepare("SELECT count(*) AS n FROM specs").get().n;
-  const rowids = (match) => new Set(db.prepare("SELECT rowid FROM specs WHERE specs MATCH ?").all(match).map((r) => r.rowid));
+function query(db, corpus, toks, { limit, exclude }) {
+  const { table } = corpus;
+  const n = db.prepare(`SELECT count(*) AS n FROM ${table}`).get().n;
+  const rowids = (match) => new Set(db.prepare(`SELECT rowid FROM ${table} WHERE ${table} MATCH ?`).all(match).map((r) => r.rowid));
   const reps = representatives(db, toks);
   const terms = [...new Set(reps.values())];
   const evidence = [];
   for (const tok of terms) {
     const all = rowids(quote(tok));
     if (all.size === 0 || all.size >= n * EVIDENCE_DF_FRACTION) continue;
-    evidence.push({ all, titleGoal: rowids(`title:${quote(tok)} OR goal:${quote(tok)}`) });
+    evidence.push({ all, strong: rowids(corpus.strong.map((f) => `${f}:${quote(tok)}`).join(" OR ")) });
   }
   if (evidence.length < MIN_EVIDENCE_TOKENS) return [];
   const rows = db.prepare(
-    `SELECT rowid, path, service, title, state,
-            bm25(specs, 0, 0, 10.0, 5.0, 2.0, 1.0, 0) AS score,
-            snippet(specs, 5, '', '', '...', 12) AS snippet
-     FROM specs WHERE specs MATCH ?`,
+    `SELECT rowid, ${corpus.select}, bm25(${table}, ${corpus.weights.map((w) => w.toFixed(1)).join(", ")}) AS score
+     FROM ${table} WHERE ${table} MATCH ?`,
   ).all(toMatch(terms)).filter((r) => !exclude.includes(r.path));
   const kept = confidenceFilter(rows, evidence);
-  kept.sort((a, b) => (a.state !== "live") - (b.state !== "live") || a.score - b.score);
-  const best = kept.find((r) => r.state === "live");
+  const live = (r) => !corpus.hasState || r.state === "live";
+  kept.sort((a, b) => Number(!live(a)) - Number(!live(b)) || a.score - b.score);
+  const best = kept.find(live);
   const cut = best ? kept.filter((r) => Math.abs(r.score) >= SCORE_RATIO * Math.abs(best.score)) : kept;
   return cut.slice(0, limit);
 }
@@ -265,20 +343,60 @@ const cell = (v) => (v === null || v === undefined ? "" : String(v).replace(/\s+
 // Paths keep their spaces; only column and row delimiters are neutralised.
 const filesCell = (v) => v.replace(/[\t\r\n]+/g, " ");
 
+function docSnippet(r) {
+  if (!r.hsnip.includes("\u0001")) return r.bsnip;
+  const fragment = r.hsnip.split("\n").find((line) => line.includes("\u0001")).replace(/[\u0001\u0002]/g, "");
+  return r.headings.split("\n").find((line) => line.includes(fragment));
+}
+
+const CORPORA = {
+  specs: {
+    table: "specs",
+    columns: ["path", "service", "title", "goal", "headings", "body", "state"],
+    strong: ["title", "goal"],
+    weights: [0, 0, 10, 5, 2, 1, 0],
+    hasState: true,
+    header: ["score", "path", "service", "title", "status", "shipped_at", "state", "files", "snippet"],
+    select: "path, service, title, state, snippet(specs, 5, '', '', '...', 12) AS snippet",
+    discover,
+    row: (text, f) => {
+      const x = extract(text, f.path);
+      return [f.path, f.service, x.title, x.goal, x.headings, text, x.state];
+    },
+    format: (root, r) => {
+      const t = telemetry(root, r.path);
+      return [...[r.score.toFixed(3), r.path, r.service, r.title, t.status, t.shipped_at, r.state].map(cell), filesCell(t.files), cell(r.snippet)];
+    },
+  },
+  docs: {
+    table: "docs",
+    columns: ["path", "title", "headings", "body"],
+    strong: ["title", "headings"],
+    weights: [0, 10, 3, 1],
+    hasState: false,
+    header: ["score", "path", "title", "snippet"],
+    select: "path, title, headings, snippet(docs, 2, '\u0001', '\u0002', '', 12) AS hsnip, snippet(docs, 3, '', '', '...', 12) AS bsnip",
+    discover: (root) => discoverDocs(root, includes(root)),
+    row: (text, f) => {
+      const x = extract(text, f.path);
+      return [f.path, x.title, x.headings, text];
+    },
+    format: (root, r) => [r.score.toFixed(3), r.path, r.title, docSnippet(r)].map(cell),
+  },
+};
+
 async function main() {
   if (!nodeOk()) die(`gauntlet-spec-index needs Node >=24.15.0 (found ${process.versions.node})`);
-  const { query: text, limit, exclude } = parseArgs(process.argv.slice(2));
+  const { query: text, corpus: name, limit, exclude } = parseArgs(process.argv.slice(2));
   const toks = tokens(text);
   if (!toks.length) usage();
   const root = repoRoot();
+  const corpus = CORPORA[name];
   const db = await openDb(root);
-  refresh(db, root, discover(root));
-  const rows = query(db, toks, { limit, exclude });
-  const out = [HEADER.join("\t")];
-  for (const r of rows) {
-    const t = telemetry(root, r.path);
-    out.push([...[r.score.toFixed(3), r.path, r.service, r.title, t.status, t.shipped_at, r.state].map(cell), filesCell(t.files), cell(r.snippet)].join("\t"));
-  }
+  refresh(db, root, corpus);
+  const rows = query(db, corpus, toks, { limit, exclude });
+  const out = [corpus.header.join("\t")];
+  for (const r of rows) out.push(corpus.format(root, r).join("\t"));
   process.stdout.write(out.join("\n") + "\n");
   db.close();
 }

package/bin/gauntlet-spec-index.test.mjs

@@ -1,6 +1,6 @@
 import { test } from "node:test";
 import assert from "node:assert/strict";
-import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, utimesSync } from "node:fs";
+import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, utimesSync, symlinkSync } from "node:fs";
 import { join, dirname } from "node:path";
 import { tmpdir } from "node:os";
 import { spawnSync } from "node:child_process";
@@ -10,6 +10,22 @@ import { DatabaseSync } from "node:sqlite";
 const CLI = join(dirname(fileURLToPath(import.meta.url)), "gauntlet-spec-index.mjs");
 const DRAFT = "# CONTEXT DRAFT - NOT A SPEC - fully replaced at spec-writing";
 const HEADER = ["score", "path", "service", "title", "status", "shipped_at", "state", "files", "snippet"];
+const DOCS_HEADER = ["score", "path", "title", "snippet"];
+const docFiller = (n) => `# Filler ${n}\n\n## Design\n\nnothing of note here\n`;
+const decoy = "# ptarmigan gannet decoy\n\n## ptarmigan gannet\n\nbody\n";
+const docsRepo = () => {
+  const root = gitRepo();
+  write(root, "doc/guide.md", "# Guide\n\n## Configuring the ptarmigan gannet\n\nprose about setup\n");
+  write(root, "README.md", "# Repo\n\nThe body mentions ptarmigan once.\n");
+  for (let n = 1; n <= 4; n++) write(root, `doc/filler-${n}.md`, docFiller(n));
+  for (const p of ["doc/specs/a.md", "docs/specs/b.md", "doc/plans/p.md", "docs/plans/q.md", "node_modules/x/doc/a.md", "build/doc/b.md", ".pi/gauntlet/n.md", ".worktrees/w/doc/g.md"]) write(root, p, decoy);
+  write(root, ".gitignore", "build/\n");
+  write(root, "doc/draft.md", `${DRAFT}\n\n## ptarmigan gannet\n`);
+  commit(root);
+  return root;
+};
+const docsPaths = (root) => withDb(root, (db) => db.prepare("SELECT path FROM docs ORDER BY path").all().map((r) => r.path));
+const overrides = (root, globs) => write(root, ".pi/gauntlet-overrides.md", `## Issue tracker\ntracker: none\n\n## Spec index\n${globs.map((g) => `- docs: \`${g}\``).join("\n")}\n\n## release\nnothing\n`);
 
 const write = (root, rel, text) => {
   mkdirSync(join(root, dirname(rel)), { recursive: true });
@@ -44,7 +60,7 @@ const repo = () => {
 const run = (cwd, args) => {
   const r = spawnSync(process.execPath, [CLI, ...args], { cwd, encoding: "utf8" });
   const lines = r.stdout.split("\n").filter(Boolean);
-  return { status: r.status, stderr: r.stderr, header: lines[0]?.split("\t"), rows: lines.slice(1).map((l) => l.split("\t")) };
+  return { status: r.status, stderr: r.stderr, stdout: r.stdout, header: lines[0]?.split("\t"), rows: lines.slice(1).map((l) => l.split("\t")) };
 };
 const paths = (res) => res.rows.map((r) => r[1]);
 const withDb = (root, fn) => {
@@ -104,7 +120,7 @@ test("4: schema_version mismatch rebuilds the db", (t) => {
   const r = run(root, ["--query", "zephyr widget"]);
   assert.equal(r.status, 0, r.stderr);
   assert.equal(paths(r).length, 2);
-  assert.equal(withDb(root, (database) => database.prepare("SELECT value FROM meta WHERE key = 'schema_version'").get().value), "2");
+  assert.equal(withDb(root, (database) => database.prepare("SELECT value FROM meta WHERE key = 'schema_version'").get().value), "3");
 });
 
 test("5: a non-SQLite database is rebuilt", (t) => {
@@ -114,7 +130,7 @@ test("5: a non-SQLite database is rebuilt", (t) => {
   const r = run(root, ["--query", "zephyr widget"]);
   assert.equal(r.status, 0, r.stderr);
   assert.deepEqual(paths(r), ["doc/specs/a.md", "svc-a/doc/specs/b.md"]);
-  assert.equal(withDb(root, (database) => database.prepare("SELECT value FROM meta WHERE key = 'schema_version'").get().value), "2");
+  assert.equal(withDb(root, (database) => database.prepare("SELECT value FROM meta WHERE key = 'schema_version'").get().value), "3");
 });
 
 test("6: telemetry join is output-only and tolerant", (t) => {
@@ -340,3 +356,205 @@ test("16: punctuation splits into distinct evidence terms and query order does n
   assert.deepEqual(paths(distinct), ["doc/specs/cfg.md"]);
   assert.deepEqual(paths(run(root, ["--query", "config config-file"])), paths(run(root, ["--query", "config-file config"])));
 });
+
+test("D1: --corpus specs and omitted --corpus are byte-identical; nine columns", (t) => {
+  const root = repo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  const plain = run(root, ["--query", "zephyr widget"]);
+  const explicit = run(root, ["--query", "zephyr widget", "--corpus", "specs"]);
+  assert.equal(explicit.status, 0, explicit.stderr);
+  assert.equal(explicit.stdout, plain.stdout);
+  assert.deepEqual(explicit.header, HEADER);
+});
+
+test("D11: unknown --corpus exits 1 with usage", (t) => {
+  const root = repo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  const r = run(root, ["--query", "zephyr widget", "--corpus", "notes"]);
+  assert.equal(r.status, 1);
+  assert.match(r.stderr, /usage:/);
+  assert.equal(run(root, ["--query", "zephyr widget", "--corpus"]).status, 1);
+});
+
+test("D12: a schema 2 cache is rebuilt to 3 with both tables present", (t) => {
+  const root = repo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  run(root, ["--query", "zephyr widget"]);
+  withDb(root, (database) => database.prepare("UPDATE meta SET value = '2' WHERE key = 'schema_version'").run());
+  const r = run(root, ["--query", "zephyr widget"]);
+  assert.equal(r.status, 0, r.stderr);
+  assert.equal(withDb(root, (database) => database.prepare("SELECT value FROM meta WHERE key = 'schema_version'").get().value), "3");
+  const tables = withDb(root, (database) => database.prepare("SELECT name FROM sqlite_master WHERE name IN ('specs', 'docs') ORDER BY name").all().map((row) => row.name));
+  assert.deepEqual(tables, ["docs", "specs"]);
+  const cols = withDb(root, (database) => database.prepare("PRAGMA table_info(files)").all().map((row) => row.name));
+  assert.deepEqual(cols, ["corpus", "path", "mtime_ms", "size"]);
+});
+
+test("D10-specs: a `## ` line inside a code fence is not a heading", (t) => {
+  const root = repo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/specs/fenced.md", "# Fenced pelican\n\n**Goal:** pelican.\n\n```markdown\n## Not a heading pelican\n```\n\n## Real heading\n");
+  run(root, ["--query", "zephyr widget"]);
+  const h = withDb(root, (database) => database.prepare("SELECT headings FROM specs WHERE path = 'doc/specs/fenced.md'").get().headings);
+  assert.equal(h, "Real heading");
+});
+
+test("D2: docs include defaults and fixed exclusions", (t) => {
+  const root = docsRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  const r = run(root, ["--corpus", "docs", "--query", "ptarmigan gannet"]);
+  assert.equal(r.status, 0, r.stderr);
+  assert.deepEqual(r.header, DOCS_HEADER);
+  assert.deepEqual(paths(r), ["doc/guide.md"]);
+  assert.equal(r.rows[0].length, 4);
+  assert.deepEqual(docsPaths(root), ["README.md", ...[1, 2, 3, 4].map((n) => `doc/filler-${n}.md`), "doc/guide.md"]);
+  overrides(root, [".pi/**/*.md", ".worktrees/**/*.md", "doc/**/*.md"]);
+  assert.equal(run(root, ["--corpus", "docs", "--query", "ptarmigan gannet"]).status, 0);
+  assert.deepEqual(docsPaths(root), [".pi/gauntlet-overrides.md", ...[1, 2, 3, 4].map((n) => `doc/filler-${n}.md`), "doc/guide.md"]);
+  const two = gitRepo();
+  t.after(() => rmSync(two, { recursive: true, force: true }));
+  write(two, "doc/x.md", decoy);
+  write(two, "doc/y.md", decoy);
+  commit(two);
+  const tiny = run(two, ["--corpus", "docs", "--query", "ptarmigan gannet"]);
+  assert.equal(tiny.status, 0, tiny.stderr);
+  assert.deepEqual(tiny.header, DOCS_HEADER);
+  assert.deepEqual(tiny.rows, []);
+});
+
+test("D3: shared confidence rejects common body-only hits", (t) => {
+  const root = gitRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/strong.md", "# Strong heron ibis\n\n## jackal\n\nbody\n");
+  for (let n = 1; n <= 5; n++) write(root, `doc/p${n}.md`, `# Plain doc ${n}\n\nThe body mentions the common shared vocabulary.\n`);
+  commit(root);
+  const r = run(root, ["--corpus", "docs", "--query", "heron ibis jackal common", "--limit", "10"]);
+  assert.equal(r.status, 0, r.stderr);
+  assert.deepEqual(paths(r), ["doc/strong.md"]);
+});
+
+test("D4: no cross-corpus leakage", (t) => {
+  const root = repo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/guide.md", "# zephyr widget guide\n\n## zephyr widget\n");
+  for (let n = 1; n <= 4; n++) write(root, `doc/filler-${n}.md`, docFiller(n));
+  assert.deepEqual(paths(run(root, ["--query", "zephyr widget"])), ["doc/specs/a.md", "svc-a/doc/specs/b.md"]);
+  const docs = run(root, ["--corpus", "docs", "--query", "zephyr widget"]);
+  assert.deepEqual(paths(docs), ["doc/guide.md"]);
+  assert.deepEqual(docs.header, DOCS_HEADER);
+});
+
+test("D5: independent cache refresh in both directions", (t) => {
+  const root = repo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/guide.md", "# Guide\n\n## zephyr widget\n");
+  run(root, ["--query", "zephyr widget"]);
+  run(root, ["--corpus", "docs", "--query", "zephyr widget"]);
+  const files = () => withDb(root, (db) => Object.fromEntries(db.prepare("SELECT corpus, path, mtime_ms FROM files").all().map((r) => [`${r.corpus}:${r.path}`, r.mtime_ms])));
+  const before = files();
+  utimesSync(join(root, "doc/specs/a.md"), new Date(), new Date(Date.now() + 5000));
+  run(root, ["--corpus", "docs", "--query", "zephyr widget"]);
+  assert.deepEqual(files(), before);
+  run(root, ["--query", "zephyr widget"]);
+  assert.notEqual(files()["specs:doc/specs/a.md"], before["specs:doc/specs/a.md"]);
+  assert.equal(files()["docs:doc/guide.md"], before["docs:doc/guide.md"]);
+  assert.deepEqual(docsPaths(root), ["doc/guide.md"]);
+});
+
+test("D6: overrides replace defaults but cannot bypass exclusions", (t) => {
+  const root = docsRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "notes/a.md", decoy);
+  write(root, "CONTRIBUTING.md", decoy);
+  write(root, "svc/CONTRIBUTING.md", decoy);
+  write(root, "svc/doc/inner.md", decoy);
+  commit(root);
+  const search = () => run(root, ["--corpus", "docs", "--query", "ptarmigan gannet"]);
+  overrides(root, ["notes/**/*.md"]);
+  assert.equal(search().status, 0);
+  assert.deepEqual(docsPaths(root), ["notes/a.md"]);
+  overrides(root, ["**/*.md"]);
+  assert.equal(search().status, 0);
+  const all = docsPaths(root);
+  for (const p of ["doc/specs/a.md", "docs/specs/b.md", "doc/plans/p.md", "docs/plans/q.md", "node_modules/x/doc/a.md", "build/doc/b.md", "doc/draft.md", ".pi/gauntlet/n.md", ".worktrees/w/doc/g.md"]) assert.ok(!all.includes(p), p);
+  assert.ok(all.includes("doc/guide.md"));
+  overrides(root, ["CONTRIBUTING.md"]);
+  assert.equal(search().status, 0);
+  assert.deepEqual(docsPaths(root), ["CONTRIBUTING.md", "svc/CONTRIBUTING.md"]);
+  write(root, ".pi/gauntlet-overrides.md", "## Spec index\n\nno bullets here\n");
+  assert.equal(search().status, 0);
+  assert.ok(docsPaths(root).includes("doc/guide.md"));
+  assert.ok(docsPaths(root).includes("svc/doc/inner.md"));
+});
+
+test("D6b: overrides file resolution order", (t) => {
+  const root = docsRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/gauntlet-overrides.md", "## Spec index\n- docs: `notes/**/*.md`\n");
+  write(root, "notes/a.md", decoy);
+  const search = () => run(root, ["--corpus", "docs", "--query", "ptarmigan gannet"]);
+  assert.equal(search().status, 0);
+  assert.deepEqual(docsPaths(root), ["notes/a.md"]);
+  write(root, "gauntlet-overrides.md", "## Spec index\n- docs: `other/**/*.md`\n");
+  write(root, "other/b.md", decoy);
+  assert.equal(search().status, 0);
+  assert.deepEqual(docsPaths(root), ["other/b.md"]);
+  overrides(root, ["doc/**/*.md"]);
+  assert.equal(search().status, 0);
+  assert.deepEqual(docsPaths(root), ["doc/filler-1.md", "doc/filler-2.md", "doc/filler-3.md", "doc/filler-4.md", "doc/gauntlet-overrides.md", "doc/guide.md"]);
+});
+
+test("D7: heading hits are strong, body-only hits are not", (t) => {
+  const root = gitRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/h.md", "# H\n\n## Setup\n\n## Feeding the kudu wombat\n\nprose\n");
+  write(root, "doc/b.md", "# B\n\n## Setup\n\nkudu wombat in the body only\n");
+  for (let n = 1; n <= 4; n++) write(root, `doc/filler-${n}.md`, docFiller(n));
+  commit(root);
+  const r = run(root, ["--corpus", "docs", "--query", "kudu wombat"]);
+  assert.equal(r.status, 0, r.stderr);
+  assert.deepEqual(paths(r), ["doc/h.md"]);
+});
+
+test("D8: snippet selects heading line or body fallback", (t) => {
+  const root = gitRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  const longHeading = "Feeding the wandering herds across twelve distant valleys before finally finding the kudu wombat";
+  write(root, "doc/h.md", `# H\n\n## Setup\n\n## ${longHeading}\n\nprose\n`);
+  write(root, "doc/t.md", "# lynx ocelot title\n\n## Unrelated\n\nbody says lynx ocelot here\n");
+  for (let n = 1; n <= 4; n++) write(root, `doc/filler-${n}.md`, docFiller(n));
+  commit(root);
+  const heading = run(root, ["--corpus", "docs", "--query", "kudu wombat"]);
+  assert.deepEqual(paths(heading), ["doc/h.md"]);
+  assert.equal(heading.rows[0][3], longHeading);
+  const body = run(root, ["--corpus", "docs", "--query", "lynx ocelot"]);
+  assert.deepEqual(paths(body), ["doc/t.md"]);
+  assert.match(body.rows[0][3], /body says lynx ocelot/);
+  assert.doesNotMatch(body.rows[0][3], /[\u0001\u0002]/);
+});
+
+test("D9: deleted files and symlinks are skipped, stale rows removed", (t) => {
+  const root = docsRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/gone.md", "# Gone\n\n## ptarmigan gannet\n");
+  commit(root);
+  run(root, ["--corpus", "docs", "--query", "ptarmigan gannet"]);
+  assert.ok(docsPaths(root).includes("doc/gone.md"));
+  rmSync(join(root, "doc/gone.md"));
+  symlinkSync(join(root, "doc/specs/a.md"), join(root, "doc/link.md"));
+  symlinkSync(join(root, "doc/nowhere.md"), join(root, "doc/dangling.md"));
+  const r = run(root, ["--corpus", "docs", "--query", "ptarmigan gannet"]);
+  assert.equal(r.status, 0, r.stderr);
+  assert.deepEqual(paths(r), ["doc/guide.md"]);
+  for (const p of ["doc/gone.md", "doc/link.md", "doc/dangling.md"]) assert.ok(!docsPaths(root).includes(p), p);
+  assert.equal(withDb(root, (db) => db.prepare("SELECT count(*) AS c FROM files WHERE corpus = 'docs' AND path = 'doc/gone.md'").get().c), 0);
+});
+
+test("D10: fenced headings and shorter nested fences are ignored", (t) => {
+  const root = docsRepo();
+  t.after(() => rmSync(root, { recursive: true, force: true }));
+  write(root, "doc/f.md", "# F\n\n```\n## kudu wombat\n```\n\nbody\n");
+  write(root, "doc/nested.md", "# N\n\n````markdown\n```js\n## inner heading\n```\n## still fenced\n````\n");
+  run(root, ["--corpus", "docs", "--query", "ptarmigan gannet"]);
+  for (const p of ["doc/f.md", "doc/nested.md"]) assert.equal(withDb(root, (db) => db.prepare("SELECT headings FROM docs WHERE path = ?").get(p).headings), "");
+});

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-gauntlet",
-  "version": "5.22.0",
+  "version": "5.23.0",
   "description": "Opinionated, gated workflow skills, subagent personas, and runtime extensions for the pi coding agent.",
   "author": "Jacek Juraszek",
   "type": "module",

package/skills/brainstorming/SKILL.md

@@ -104,7 +104,7 @@ Prefer clear testable boundaries, YAGNI, existing conventions, the owning schema
 
 ### 6. Present the design in two rounds
 
-Use two rounds, targeting 300-500 words each, with one approval each; revisions remain within that approval point. Ask once per round; round-1 approval without correction confirms the predecessor. Round 1 covers architecture, responsibilities, data flow, and `supersedes <path>, <scope>` when applicable. Round 2 covers errors, edges, tests, and `## Documentation impact`.
+Use two rounds, targeting 300-500 words each, with one approval each; revisions remain within that approval point. Ask once per round; round-1 approval without correction confirms the predecessor. Round 1 covers architecture, responsibilities, data flow, and `supersedes <path>, <scope>` when applicable. Round 2 covers errors, edges, tests, and `## Documentation impact`. In round 2, cite the draft's `Docs touched:` entries as candidates for "Materially amended existing docs"; admit or drop each by the materiality bar in `reference/documentation-impact.md`, never list them automatically.
 
 `## Documentation impact` is required. Cite `reference/documentation-impact.md` by relative path, do not restate its categories, and reproduce this template verbatim:
 

package/skills/brainstorming/gatherer.md

@@ -82,7 +82,22 @@ Scout (always dispatched):
 > from the directory-listing fallback) contributes no path and no `missing` line. When
 > no path and no `missing` line results - including whenever the index was unavailable -
 > omit the section entirely; `Predecessor: none` produces no anchors section. The anchors
-> are a recon hint, never a selection input. End with an
+> are a recon hint, never a selection input.
+> Docs check: run `node <SPEC_INDEX> --corpus docs --query '<the same keywords>' --limit 10`
+> from the worktree root. Its rows are project documentation that already speaks about the
+> request's topic; open at most five whose topic matches and let them inform the "already
+> solves this?" finding and the current-contract recon. If the command fails, write
+> `Docs index unavailable - docs check used recon only.` in your handoff and continue. After
+> the `Predecessor:` line(s) and any `Predecessor anchors` section, render at most
+> five `Docs touched: <path> - <section heading>` lines, one per document that
+> genuinely covers the request's topic - an opened index row or a document your own
+> recon found - where
+> `<section heading>` is the `##`/`###` heading of the covering section as read in the
+> document (the row's `snippet` column is a hint to it), or the document title when no single
+> section applies; or `Docs touched: none`. Judge by topic; a lexical hit alone is not
+> coverage. Zero rows means the index found no evidence, not that no document covers the
+> topic. This query supplements the code and documentation recon you already perform; it
+> never replaces it - keep reading the files the request touches. End with an
 > "Open questions that matter for the spec"
 > section. Compact handoff, not a dump.