@skyphusion/search-mcp 0.2.0 → 0.3.0

package/README.md

@@ -108,6 +108,89 @@ npm run sync:run     # isolated clone root, sync all targets, optional reindex
 
 The sync remaps non-native extensions (`.ts`, `.tsx`, extensionless `Dockerfile`, `.service`, etc.) to `.txt` keys so AI Search indexes them. See `scripts/sync-ingest.mjs`.
 
+### Bounding a corpus: `includePaths` vs `excludePaths`
+
+`excludePaths` is a denylist and is **fail-open**: add a new top-level file to a
+repo and it silently joins the corpus. That is fine for a docs site. It is wrong
+whenever the corpus boundary actually matters, because "we forgot to exclude it"
+becomes a real incident.
+
+`includePaths` is an allowlist and is **fail-closed**: when a repo has an entry,
+only paths under those prefixes are eligible and everything else is refused.
+
+```json
+{
+  "includePaths": { "my-repo": ["docs/", "_corpus/"] },
+  "excludePaths": { "my-repo": ["docs/internal/"] }
+}
+```
+
+The two compose subtractively: `includePaths` decides what is eligible, then
+`excludePaths` subtracts from that. A denylist entry can never add a path back.
+
+Omit a repo from `includePaths` to keep the previous behaviour. An entry that is
+present but **empty** (`[]`) is a configuration error, not a way to say "index
+nothing": those are different states and collapsing them into the permissive one
+is how a whole repo joins a corpus by accident.
+
+#### A present allowlist must earn its keep
+
+An allowlist that quietly matches nothing is worse than no allowlist. The sync
+plans zero objects, the mirror prune deletes the corpus that was there, the
+reindex succeeds over nothing, and the answer surface returns a confident nothing
+with every status light green. So each of these refuses the run (exit 2) before
+anything uploads or is pruned:
+
+| Refusal | Meaning |
+| --- | --- |
+| `include_paths_no_match` | An entry matched zero git-tracked files (a typo, or a directory that moved) |
+| `include_paths_entry_empty` | The entry is `[]` |
+| `include_paths_entry_invalid` | An entry cannot match a git path (absolute, `..`, backslash, non-string) |
+| `include_paths_entry_not_array` / `include_paths_not_object` | Wrong config shape |
+| `include_paths_unknown_repo` | The entry names a repo no target lists, so nothing reads it |
+| `include_paths_repo_not_cloned` | An allowlisted repo is missing from the clone root |
+| `include_paths_all_excluded` | `excludePaths` removed everything the allowlist selected |
+| `include_paths_all_filtered` | Nothing the allowlist selected survived the ingest filters |
+
+Shape and repo-name checks run for **every** target on every sync, not just the
+one being synced, so a rule that has rotted is caught by the next run of any
+target. A stale `excludePaths` repo name warns rather than refuses: denylist rot
+grows a corpus, allowlist rot empties one.
+
+Verify what a target will actually upload before you trust it:
+
+```sh
+node scripts/sync.mjs my-target --dry-run
+```
+
+### Size cap
+
+Objects over `SYNC_MAX_BYTES` (default 4 MB) are skipped. Skips are summarised at
+the end of the run, not only warned inline, because a file silently missing from
+the corpus looks exactly like a file the corpus does not contain -- the worst
+failure mode for something that answers questions. Pass `--fail-on-skip` to turn
+an incomplete corpus into a failed run.
+
+```sh
+SYNC_MAX_BYTES=$((16 * 1024 * 1024)) node scripts/sync.mjs corpus --fail-on-skip
+```
+
+### Corpus manifest (optional)
+
+A corpus producer can publish a `manifest.json` next to its objects so the widget
+renders citations instead of raw R2 keys. Entries need a `key`; `title`, `url`,
+`page`, and `total_pages` are used when present:
+
+```json
+{ "pages": [
+  { "key": "my-doc/p003.txt", "title": "Deploy guide", "url": "/docs/deploy/", "page": 3, "total_pages": 12 }
+] }
+```
+
+Keys are matched exactly first, then by **suffix**, because the sync namespaces
+every object under its repo name (`<repo>/<path>`) while a producer naturally
+writes its manifest in terms of its own paths.
+
 ### Reindex dispatch
 
 AI Search rejects a new reindex job for two distinct reasons, and `sync-runner` clears both
@@ -144,9 +227,30 @@ Copy `public/ask-widget.js` and `public/ask-widget.css` to your docs site:
         data-endpoint="https://search.example.com/ask"
         data-target="#docs-ask"
         data-label="Ask the docs"
+        data-manifest="/corpus-manifest.json"
+        data-empty-text="Nothing in the indexed corpus addresses that."
         data-sitekey="YOUR_TURNSTILE_SITEKEY"></script>
 ```
 
+`data-manifest` is optional; without it sources render as raw object keys. A
+missing or malformed manifest degrades to raw keys rather than breaking answers.
+
+`data-empty-text` is shown when a query returns no retrieved sources, so an
+unsourced answer is never left on screen looking authoritative.
+
+### Per-site system prompts
+
+One deployment can serve several sites. `ORIGIN_PROFILES` is a JSON object in
+`[vars]` mapping an exact request `Origin` to that site's system prompt:
+
+```toml
+ORIGIN_PROFILES = '{"https://docs.example.com":"You are the docs assistant...","https://other.example":"You are..."}'
+```
+
+Precedence is `ORIGIN_PROFILES`, then the legacy blog special-case, then
+`ASSISTANT_SYSTEM_PROMPT`. A malformed value is logged and ignored so a bad var
+cannot take `/ask` down.
+
 ## Who this is for
 
 Operators building documentation search, agent tooling, or internal knowledge bases on [Cloudflare AI Search](https://developers.cloudflare.com/ai-search/) with MCP and a browser widget.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@skyphusion/search-mcp",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Bearer-gated MCP server and streaming query Worker over Cloudflare AI Search, plus a git-to-R2 corpus sync toolkit.",
   "license": "AGPL-3.0-only",
   "author": "Conrad Rockenhaus <conrad@skyphusion.org>",
@@ -60,7 +60,9 @@
     "sync:internal": "node scripts/sync.mjs internal",
     "sync:dry": "node scripts/sync.mjs corpus --dry-run",
     "sync:run": "node scripts/sync-runner.mjs",
-    "materialize-config": "node scripts/materialize-config.mjs"
+    "materialize-config": "node scripts/materialize-config.mjs",
+    "escrow": "node scripts/escrow-targets.mjs",
+    "guard:targets": "node scripts/guard-targets-additive.mjs"
   },
   "dependencies": {
     "@aws-sdk/client-s3": "^3.888.0"

package/public/ask-widget.js

@@ -16,6 +16,12 @@
   var label = script.getAttribute("data-label") || "Ask";
   var placeholder = script.getAttribute("data-placeholder") || "Ask a question about the docs";
   var sitekey = script.getAttribute("data-sitekey") || "";
+  // Optional corpus manifest: maps an R2 object key to a human title, a URL,
+  // and a page number, so sources render as citations instead of raw keys.
+  var manifestUrl = script.getAttribute("data-manifest") || "";
+  var emptyText =
+    script.getAttribute("data-empty-text") ||
+    "Nothing in the indexed corpus addresses that.";
 
   function ready(fn) {
     if (document.readyState !== "loading") fn();
@@ -69,6 +75,97 @@
       }
     }
 
+
+    // ---- corpus manifest -------------------------------------------------
+    //
+    // Without a manifest the widget can only show the R2 object key, which is
+    // an implementation detail ("repo/docs/DEPLOY.md"). With one it can show
+    // what a reader actually needs: the document title, a link, and a page.
+    //
+    // Entries are matched by exact key first, then by SUFFIX. Suffix matching
+    // matters because the sync prefixes every object with its repo name, so the
+    // key in R2 is "<repo>/<path>" while a corpus producer naturally writes its
+    // manifest in terms of its own paths. Suffix matching lets the producer stay
+    // ignorant of how the sync namespaces things.
+    var manifest = null;
+    var manifestIndex = null;
+
+    function indexManifest(data) {
+      var entries = [];
+      if (Array.isArray(data)) entries = data;
+      else if (data && Array.isArray(data.pages)) entries = data.pages;
+      else if (data && Array.isArray(data.objects)) entries = data.objects;
+      var byKey = {};
+      entries.forEach(function (e) {
+        if (e && typeof e.key === "string") byKey[e.key] = e;
+      });
+      return { byKey: byKey, entries: entries };
+    }
+
+    function loadManifest() {
+      if (!manifestUrl) return Promise.resolve(null);
+      if (manifest) return Promise.resolve(manifest);
+      return fetch(manifestUrl)
+        .then(function (r) {
+          return r.ok ? r.json() : null;
+        })
+        .then(function (data) {
+          if (!data) return null;
+          manifest = data;
+          manifestIndex = indexManifest(data);
+          return data;
+        })
+        .catch(function () {
+          // A missing or broken manifest must not break answers. Degrade to
+          // showing raw keys rather than failing the request.
+          return null;
+        });
+    }
+
+    function lookup(key) {
+      if (!manifestIndex) return null;
+      if (manifestIndex.byKey[key]) return manifestIndex.byKey[key];
+      for (var i = 0; i < manifestIndex.entries.length; i++) {
+        var e = manifestIndex.entries[i];
+        if (e && typeof e.key === "string" && key.indexOf(e.key) >= 0) {
+          var at = key.length - e.key.length;
+          if (at >= 0 && key.slice(at) === e.key) return e;
+        }
+      }
+      return null;
+    }
+
+    function citationNode(key) {
+      var entry = lookup(key);
+      var li = document.createElement("li");
+      if (!entry) {
+        li.textContent = key;
+        return li;
+      }
+      var label = entry.title || key;
+      if (entry.url) {
+        var a = document.createElement("a");
+        a.setAttribute("href", entry.url);
+        a.textContent = label;
+        li.appendChild(a);
+      } else {
+        li.appendChild(document.createTextNode(label));
+      }
+      var bits = [];
+      if (entry.page) {
+        bits.push("page " + entry.page + (entry.total_pages ? " of " + entry.total_pages : ""));
+      }
+      if (entry.case_number) bits.push("case " + entry.case_number);
+      if (entry.filed_date) bits.push("filed " + entry.filed_date);
+      if (bits.length) {
+        var span = document.createElement("span");
+        span.className = "vjask-source-meta";
+        span.textContent = " (" + bits.join(", ") + ")";
+        li.appendChild(span);
+      }
+      return li;
+    }
+
     function renderSources(chunks) {
       var seen = {};
       var items = [];
@@ -80,12 +177,18 @@
         }
       });
       if (!items.length) return;
-      sources.innerHTML = "<li class='vjask-sources-title'>Sources</li>" +
-        items.map(function (k) {
-          return "<li>" + k.replace(/[&<>]/g, function (ch) {
-            return { "&": "&amp;", "<": "&lt;", ">": "&gt;" }[ch];
-          }) + "</li>";
-        }).join("");
+      sawSources = true;
+      // Built via the DOM API rather than innerHTML: entries come from the
+      // corpus manifest, and concatenating them into markup would be an
+      // injection path through a data file.
+      sources.textContent = "";
+      var title = document.createElement("li");
+      title.className = "vjask-sources-title";
+      title.textContent = "Sources";
+      sources.appendChild(title);
+      items.forEach(function (k) {
+        sources.appendChild(citationNode(k));
+      });
       sources.hidden = false;
     }
 
@@ -109,6 +212,8 @@
       if (delta && delta.content) answer.textContent += delta.content;
     }
 
+    var sawSources = false;
+
     form.addEventListener("submit", function (ev) {
       ev.preventDefault();
       var q = input.value.trim();
@@ -116,8 +221,10 @@
       btn.disabled = true;
       answer.textContent = "";
       sources.hidden = true;
-      sources.innerHTML = "";
+      sources.textContent = "";
+      sawSources = false;
 
+      loadManifest().then(function () {
       fetch(endpoint, {
         method: "POST",
         headers: { "content-type": "application/json" },
@@ -153,6 +260,12 @@
           answer.textContent = "Sorry, something went wrong (" + err.message + ").";
         })
         .finally(function () {
+          // No retrieved sources means nothing in the corpus matched. Say that
+          // outright rather than leaving a plausible-looking unsourced answer
+          // on screen.
+          if (!sawSources && !answer.textContent.trim()) {
+            answer.textContent = emptyText;
+          }
           btn.disabled = false;
           if (sitekey && window.turnstile) {
             try {
@@ -160,6 +273,7 @@
             } catch (e) {}
           }
         });
+      });
     });
   });
 })();

package/scripts/sync-ingest.mjs

@@ -123,19 +123,253 @@ export function ingestKind(relPath) {
   return DOC_EXT.has(ext) || basename(relPath).startsWith("README") ? "doc" : "code";
 }
 
+
 /**
- * Per-repo corpus exclusion (targets.json `excludePaths`). An entry is a repo-relative
- * path: with a trailing "/" it excludes the whole subtree; without, it excludes that
- * exact file or the subtree rooted at that name.
+ * One prefix rule, shared by the allowlist and the denylist so the two can
+ * never drift. An entry is a repo-relative path: with a trailing "/" it means
+ * the subtree, without one it means that exact file or the subtree rooted at
+ * that name (so "docs" matches "docs" and "docs/a.md" but never "docsite/a.md").
+ */
+export function matchesPathPrefix(relPath, prefix) {
+  if (typeof prefix !== "string" || prefix === "") return false;
+  if (prefix.endsWith("/")) return relPath.startsWith(prefix);
+  return relPath === prefix || relPath.startsWith(prefix + "/");
+}
+
+/**
+ * Per-repo corpus exclusion (targets.json `excludePaths`). Denylist, applied on
+ * top of the allowlist. Absent or empty means "exclude nothing".
  */
 export function isExcludedPath(relPath, prefixes) {
   for (const p of prefixes || []) {
-    if (!p) continue;
-    if (p.endsWith("/")) {
-      if (relPath.startsWith(p)) return true;
-    } else if (relPath === p || relPath.startsWith(p + "/")) {
-      return true;
-    }
+    if (matchesPathPrefix(relPath, p)) return true;
+  }
+  return false;
+}
+
+/**
+ * Per-repo corpus ALLOWLIST (targets.json `includePaths`). When a repo has an
+ * entry, ONLY paths under one of its prefixes are eligible for the corpus;
+ * everything else is refused. When a repo has NO entry (undefined/null) every
+ * path is eligible and `excludePaths` alone applies, so a config that never
+ * mentions includePaths behaves exactly as it did before.
+ *
+ * Why both exist. A denylist is fail-OPEN: add a new top-level file to a repo
+ * and it silently joins the corpus. That is fine for a docs site and wrong for
+ * a corpus whose boundary matters -- a court-record mirror, anything with a
+ * privileged directory, anything where "we forgot to exclude it" is an incident
+ * rather than noise. An allowlist is fail-CLOSED: a new path is ineligible
+ * until someone says otherwise.
+ *
+ * An entry that is PRESENT but empty (or malformed) matches nothing rather than
+ * everything. "The operator wrote an allowlist" and "the operator wrote no
+ * allowlist" are different states and must not collapse into the permissive
+ * one; the config validation below turns that state into a hard error before a
+ * sync ever reaches this matcher.
+ */
+export function isIncludedPath(relPath, prefixes) {
+  if (prefixes === undefined || prefixes === null) return true;
+  if (!Array.isArray(prefixes)) return false;
+  for (const p of prefixes) {
+    if (matchesPathPrefix(relPath, p)) return true;
   }
   return false;
 }
+
+/**
+ * Thrown for every includePaths refusal. `code` names the specific state so a
+ * test can prove a failure happened for the stated reason and not by accident.
+ */
+export class IncludePathsError extends Error {
+  constructor(code, message) {
+    super(message);
+    this.name = "IncludePathsError";
+    this.code = code;
+  }
+}
+
+function describeType(v) {
+  if (v === null) return "null";
+  if (Array.isArray(v)) return "array";
+  return typeof v;
+}
+
+function q(v) {
+  return typeof v === "string" ? JSON.stringify(v) : describeType(v) + " " + JSON.stringify(v);
+}
+
+/** Every repo named by any target. An includePaths key outside this set reaches nothing. */
+export function knownTargetRepos(cfg) {
+  const repos = new Set();
+  for (const target of Object.values(cfg?.targets || {})) {
+    for (const repo of target?.repos || []) repos.add(repo);
+  }
+  return repos;
+}
+
+/** Shape check for one repo entry. Returns {code, message} objects, never throws. */
+export function validateIncludePathsEntry(repo, prefixes) {
+  const errors = [];
+  if (!Array.isArray(prefixes)) {
+    errors.push({
+      code: "include_paths_entry_not_array",
+      message:
+        `includePaths for repo ${q(repo)} must be an array of repo-relative path prefixes, ` +
+        `got ${describeType(prefixes)}. Shape: "includePaths": { "my-repo": ["_corpus/"] }.`,
+    });
+    return errors;
+  }
+  if (prefixes.length === 0) {
+    errors.push({
+      code: "include_paths_entry_empty",
+      message:
+        `includePaths for repo ${q(repo)} is an empty array, which is a configuration error, ` +
+        `not an instruction to index nothing. Remove the repo entry to index all of it, or ` +
+        `list the prefixes to index.`,
+    });
+    return errors;
+  }
+  for (const p of prefixes) {
+    if (typeof p !== "string" || p.trim() === "") {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: entries must be non-empty strings.`,
+      });
+    } else if (p.startsWith("/")) {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: entries are repo-relative and ` +
+          `must not start with "/". A git path never does, so this would match nothing.`,
+      });
+    } else if (p.split("/").includes("..")) {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: ".." is not something a git path ` +
+          `can start with, so this would match nothing.`,
+      });
+    } else if (p.includes("\\")) {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: git paths use "/" as the ` +
+          `separator, so a backslash would match nothing.`,
+      });
+    }
+  }
+  return errors;
+}
+
+/**
+ * Whole-config check for targets.json `includePaths`. Returns {code, message}
+ * objects, never throws, so a caller can report all of them at once.
+ *
+ * Rejects an entry naming a repo no target lists: a rule nothing reads is
+ * indistinguishable from a rule that works, which is how an allowlist rots.
+ */
+export function validateIncludePathsConfig(cfg) {
+  const errors = [];
+  const includePaths = cfg?.includePaths;
+  if (includePaths === undefined || includePaths === null) return errors;
+  if (typeof includePaths !== "object" || Array.isArray(includePaths)) {
+    errors.push({
+      code: "include_paths_not_object",
+      message:
+        `targets.json includePaths must be an object keyed by repo, got ` +
+        `${describeType(includePaths)}. Shape: "includePaths": { "my-repo": ["_corpus/"] }.`,
+    });
+    return errors;
+  }
+  const known = knownTargetRepos(cfg);
+  for (const [repo, prefixes] of Object.entries(includePaths)) {
+    if (!known.has(repo)) {
+      errors.push({
+        code: "include_paths_unknown_repo",
+        message:
+          `includePaths names repo ${q(repo)}, which no target lists in its repos. Nothing reads ` +
+          `that rule, and a rule nothing reads is indistinguishable from a rule that works. Add ` +
+          `the repo to a target or delete the entry. Repos across all targets: ` +
+          `${[...known].join(", ") || "(none)"}.`,
+      });
+    }
+    errors.push(...validateIncludePathsEntry(repo, prefixes));
+  }
+  return errors;
+}
+
+/** Fail-closed wrapper: throws IncludePathsError carrying the first code and every message. */
+export function assertIncludePathsConfig(cfg) {
+  const errors = validateIncludePathsConfig(cfg);
+  if (!errors.length) return;
+  throw new IncludePathsError(errors[0].code, errors.map((e) => e.message).join(" "));
+}
+
+/**
+ * excludePaths keys naming a repo no target lists. Denylist rot is quieter than
+ * allowlist rot (the corpus grows rather than empties), so this is a warning at
+ * the call site, not a refusal.
+ */
+export function unknownExcludePathsRepos(cfg) {
+  const excludePaths = cfg?.excludePaths;
+  if (!excludePaths || typeof excludePaths !== "object" || Array.isArray(excludePaths)) return [];
+  const known = knownTargetRepos(cfg);
+  return Object.keys(excludePaths).filter((repo) => !known.has(repo));
+}
+
+/**
+ * Pure path selection for one repo: allowlist first, then denylist on top, so
+ * the two compose as defense in depth rather than one replacing the other.
+ *
+ * `relPaths` is the repo git-tracked file list. Refuses, loudly:
+ *
+ *   include_paths_entry_*       the entry is malformed or empty
+ *   include_paths_no_match      an entry matched ZERO tracked files
+ *   include_paths_all_excluded  the allowlist matched, excludePaths took it all
+ *
+ * The zero-match refusal is the point of the whole mechanism. A prefix that
+ * matches nothing (a rename, a misspelling, a directory that moved) otherwise
+ * plans an empty corpus, the mirror prune deletes what was there, the reindex
+ * succeeds over nothing, and the answer surface returns a confident nothing with
+ * every status light green. Coverage is measured against the tracked list before
+ * exclusion, so "this prefix names nothing in the repo" stays a distinct
+ * diagnostic from "everything it named was excluded".
+ */
+export function selectRepoPaths(repo, relPaths, opts = {}) {
+  const { includePrefixes, excludePrefixes } = opts;
+  const paths = relPaths || [];
+  if (includePrefixes === undefined || includePrefixes === null) {
+    return paths.filter((rel) => !isExcludedPath(rel, excludePrefixes));
+  }
+
+  const shapeErrors = validateIncludePathsEntry(repo, includePrefixes);
+  if (shapeErrors.length) {
+    throw new IncludePathsError(shapeErrors[0].code, shapeErrors.map((e) => e.message).join(" "));
+  }
+
+  const unmatched = includePrefixes.filter((p) => !paths.some((rel) => matchesPathPrefix(rel, p)));
+  if (unmatched.length) {
+    throw new IncludePathsError(
+      "include_paths_no_match",
+      `includePaths for repo ${q(repo)} matched 0 git-tracked files: ` +
+        `${unmatched.map(q).join(", ")} (the repo has ${paths.length} tracked file(s)). ` +
+        `An allowlist entry that matches nothing would sync an empty corpus over a healthy one ` +
+        `and report success, so the sync refuses instead. Fix the prefix in targets.json ` +
+        `(repo-relative, trailing slash for a subtree), or drop the entry if the path is gone. ` +
+        `Only the prefixes named above matched nothing.`,
+    );
+  }
+
+  const included = paths.filter((rel) => isIncludedPath(rel, includePrefixes));
+  const kept = included.filter((rel) => !isExcludedPath(rel, excludePrefixes));
+  if (kept.length === 0) {
+    throw new IncludePathsError(
+      "include_paths_all_excluded",
+      `includePaths for repo ${q(repo)} selected ${included.length} file(s) and excludePaths ` +
+        `removed every one of them. That leaves nothing to sync for this repo, which is an ` +
+        `empty corpus dressed as a successful run. Widen includePaths or narrow excludePaths.`,
+    );
+  }
+  return kept;
+}

package/scripts/sync.mjs

@@ -36,7 +36,10 @@ import {
   ingestContentType,
   ingestKind,
   fileExt,
-  isExcludedPath,
+  selectRepoPaths,
+  assertIncludePathsConfig,
+  unknownExcludePathsRepos,
+  IncludePathsError,
 } from "./sync-ingest.mjs";
 import {
   resolveTargetsPath,
@@ -47,11 +50,18 @@ import {
 const HERE = dirname(fileURLToPath(import.meta.url));
 const TARGETS_PATH = resolveTargetsPath(HERE);
 const REPO_ROOT = defaultRepoRoot(HERE, TARGETS_PATH);
-const MAX_BYTES = 4 * 1024 * 1024;
+// Objects larger than this are skipped. Configurable because the right value
+// depends on the corpus: 4 MB is fine for a source tree and wrong for scanned
+// PDFs, where the single largest document is often the one that matters most.
+// Skips are also summarised at the end of the run, not only warned inline: a
+// warning in a long CI log is a silent omission in practice.
+const MAX_BYTES = Number(process.env.SYNC_MAX_BYTES || 4 * 1024 * 1024);
 
 const args = process.argv.slice(2);
 const DRY = args.includes("--dry-run");
 const SKIP_GITHUB = args.includes("--no-github-verify") || process.env.SYNC_SKIP_GITHUB_VERIFY === "1";
+// Opt-in strictness: turn an incomplete corpus into a failed run.
+const FAIL_ON_SKIP = args.includes("--fail-on-skip");
 const targetName = args.find((a) => !a.startsWith("--"));
 
 const SKIP_EXT = new Set([
@@ -123,16 +133,33 @@ function loadConfig() {
   return JSON.parse(readFileSync(TARGETS_PATH, "utf8"));
 }
 
-function planRepo(repo, excludePrefixes = []) {
+function planRepo(repo, excludePrefixes, includePrefixes, skipped = []) {
   const repoDir = join(REPO_ROOT, repo);
+  const hasAllowlist = includePrefixes !== undefined && includePrefixes !== null;
   if (!existsSync(repoDir)) {
+    // A repo with an allowlist has been described exactly: it contributes these
+    // paths and nothing else. A missing clone contributes nothing, which is the
+    // same empty-corpus-with-green-lights failure the allowlist exists to stop.
+    if (hasAllowlist) {
+      throw new IncludePathsError(
+        "include_paths_repo_not_cloned",
+        `includePaths for repo "${repo}" cannot be enforced: not cloned at ${repoDir}. ` +
+          `A repo with an allowlist is expected to contribute exactly those paths, so a ` +
+          `missing clone refuses the sync instead of quietly syncing nothing for it.`,
+      );
+    }
     console.warn(`  ! skip ${repo}: not cloned at ${repoDir}`);
     return [];
   }
+  // Allowlist first, then denylist on top; both refusals happen here, before a
+  // single object is uploaded or pruned.
+  const candidates = selectRepoPaths(repo, trackedFiles(repoDir), {
+    includePrefixes,
+    excludePrefixes,
+  });
   const items = [];
-  for (const rel of trackedFiles(repoDir)) {
+  for (const rel of candidates) {
     if (shouldSkip(rel)) continue;
-    if (isExcludedPath(rel, excludePrefixes)) continue;
     const abs = join(repoDir, rel);
     let size;
     try {
@@ -141,7 +168,10 @@ function planRepo(repo, excludePrefixes = []) {
       continue;
     }
     if (size > MAX_BYTES) {
-      console.warn(`  ! skip ${repo}/${rel}: ${(size / 1048576).toFixed(1)} MB over 4 MB`);
+      const mb = (size / 1048576).toFixed(1);
+      const capMb = (MAX_BYTES / 1048576).toFixed(1);
+      console.warn(`  ! skip ${repo}/${rel}: ${mb} MB over the ${capMb} MB cap`);
+      skipped.push({ repo, path: rel, mb });
       continue;
     }
     const sample = readTextSample(abs, size);
@@ -161,12 +191,23 @@ function planRepo(repo, excludePrefixes = []) {
       },
     });
   }
+  if (hasAllowlist && items.length === 0) {
+    throw new IncludePathsError(
+      "include_paths_all_filtered",
+      `includePaths for repo "${repo}" selected ${candidates.length} file(s) and none of them ` +
+        `survived the ingest filters (build/vendor directories, lockfiles, .env, binaries, the ` +
+        `size cap, or the text sniff). Nothing would upload for this repo, so the sync refuses ` +
+        `rather than publish an empty corpus.`,
+    );
+  }
   return items;
 }
 
 async function main() {
   if (!targetName) {
-    console.error("usage: node scripts/sync.mjs <target> [--dry-run] [--no-github-verify]");
+    console.error(
+      "usage: node scripts/sync.mjs <target> [--dry-run] [--no-github-verify] [--fail-on-skip]",
+    );
     process.exit(2);
   }
   const cfg = loadConfig();
@@ -176,6 +217,17 @@ async function main() {
     process.exit(2);
   }
 
+  // Config-level allowlist checks run for every target, not just the one being
+  // synced: a rule that names a repo no target lists is rot, and rot found only
+  // when someone syncs that one target is rot found late.
+  assertIncludePathsConfig(cfg);
+  for (const repo of unknownExcludePathsRepos(cfg)) {
+    console.warn(
+      `  ! excludePaths names repo '${repo}', which no target lists in its repos. ` +
+        "That rule is inert; delete it or add the repo to a target.",
+    );
+  }
+
   assertPublicCorpusBoundary(cfg, targetName);
   if (targetName === "public" && !SKIP_GITHUB) {
     const gh = await assertPublicGithubVisibility(cfg);
@@ -195,13 +247,35 @@ async function main() {
   console.log(`Repo root: ${REPO_ROOT}${DRY ? "  [DRY RUN]" : ""}`);
 
   const plan = [];
+  const skipped = [];
   for (const repo of target.repos) {
-    const items = planRepo(repo, cfg.excludePaths?.[repo]);
+    const items = planRepo(
+      repo,
+      cfg.excludePaths?.[repo],
+      cfg.includePaths?.[repo],
+      skipped,
+    );
     if (items.length) console.log(`  + ${repo}: ${items.length} files`);
     plan.push(...items);
   }
   console.log(`Planned ${plan.length} objects for ${target.bucket}.`);
 
+  // Summarise oversize skips. A file silently missing from the corpus looks
+  // exactly like a file the corpus does not contain, which is the worst
+  // possible failure mode for something that answers questions.
+  if (skipped.length) {
+    console.warn(`\n  ! ${skipped.length} file(s) skipped for exceeding the size cap:`);
+    for (const s of skipped) console.warn(`      ${s.repo}/${s.path} (${s.mb} MB)`);
+    console.warn(
+      "    These are NOT in the corpus. Raise SYNC_MAX_BYTES, extract their text into\n" +
+        "    a smaller form, or accept that nothing can answer from them.",
+    );
+    if (FAIL_ON_SKIP) {
+      console.error("--fail-on-skip: refusing to continue with an incomplete corpus.");
+      process.exit(1);
+    }
+  }
+
   if (DRY) {
     for (const it of plan.slice(0, 25)) console.log(`    ${it.key}`);
     if (plan.length > 25) console.log(`    ... and ${plan.length - 25} more`);
@@ -283,6 +357,12 @@ async function main() {
 }
 
 main().catch((err) => {
+  // A config refusal is an operator error, not a crash: print the diagnostic
+  // without a stack, and exit 2 like the other targets.json failures.
+  if (err instanceof IncludePathsError) {
+    console.error(`\n${err.message}\n  [${err.code}]`);
+    process.exit(2);
+  }
   console.error(err);
   process.exit(1);
 });

package/scripts/targets.json.example

@@ -1,5 +1,11 @@
 {
   "_comment": "Copy to targets.json (./targets.json for npm CLI, or scripts/targets.json in a git clone). repos are directory names under SYNC_REPO_ROOT (default: cwd for npm, parent of this repo for dev clone). Only git-tracked files are uploaded.",
+  "_includePaths_comment": "ALLOWLIST, optional, per repo. When present, ONLY paths under these prefixes are eligible for the corpus and everything else is refused. Use it when the corpus boundary matters: a denylist is fail-OPEN, so a new top-level file silently joins the corpus, whereas an allowlist is fail-CLOSED. Omit a repo to keep the old behaviour of admitting everything; an entry that is present but EMPTY is an error, not a way to index nothing.",
+  "_includePaths_refusals": "A present allowlist must earn its keep: an entry matching zero git-tracked files, an entry naming a repo no target lists, a repo whose clone is missing, or a selection that excludePaths or the ingest filters empty out all refuse the sync (exit 2). An allowlist that quietly matches nothing would prune a healthy corpus and reindex over the emptiness with every status light green.",
+  "includePaths": {
+    "example-repo": ["docs/"]
+  },
+  "_excludePaths_comment": "DENYLIST, optional, per repo. Applied after includePaths.",
   "excludePaths": {
     "example-repo": ["vendor/private-notes/"]
   },