@skyphusion/search-mcp 0.2.1 → 0.4.0

package/README.md

@@ -48,7 +48,7 @@ cp wrangler.mcp.toml.example wrangler.mcp.toml
 cp scripts/targets.json.example scripts/targets.json
 # edit the three files for your account, instance, bucket, and repos
 
-npm run typecheck
+npm run typecheck   # src + index.test.ts (Workers types) AND scripts/**/*.test.ts (Node types)
 npm test
 ```
 
@@ -56,25 +56,34 @@ Provision R2 + AI Search, sync your corpus, deploy both Workers. Step-by-step: [
 
 ## Skyphusion production
 
-This repo is the production home for Skyphusion AI Search (`search.vivijure.com`,
-`search-internal.vivijure.com`). Config is materialized from GitHub Actions secrets at deploy/sync
-time; do not commit `wrangler.toml`, `wrangler.mcp.toml`, or `scripts/targets.json`.
+This repo is the production home for three AI Search surfaces:
 
-- [Operator runbook](docs/skyphusion/OPERATOR.md) -- secrets, topology, bootstrap
-- [Cutover record (2026-07-08)](docs/skyphusion/CUTOVER.md) -- migration history, failure log, archive steps
+| Surface | Host | Worker config |
+| --- | --- | --- |
+| Public docs / marketing | `search.vivijure.com` | secret `SKYPHUSION_WRANGLER_TOML` → `wrangler.toml` at CI |
+| Internal MCP | `search-internal.vivijure.com` | secret `SKYPHUSION_WRANGLER_MCP_TOML` |
+| Rockenhaus court records | `search.rockenhaus.net` | committed `wrangler.rockenhaus.toml` |
+
+Do not commit `wrangler.toml`, `wrangler.mcp.toml`, or `scripts/targets.json` (gitignored).
+Rockenhaus is the exception: its wrangler file is public and tracked.
+
+- [Operator runbook](docs/skyphusion/OPERATOR.md) -- secrets, topology shape, escrow, reindex
+- [Cutover record (2026-07-08)](docs/skyphusion/CUTOVER.md) -- migration history (historical)
 
 ## Workers
 
 | Worker | Entry | Endpoint | Auth |
 | --- | --- | --- | --- |
-| Query | `wrangler.toml` | `POST /ask`, `GET /health` | Turnstile (optional) + CORS allowlist |
-| MCP | `wrangler.mcp.toml` | `POST /mcp`, `GET /health` | `Authorization: Bearer` (fail closed) |
+| Query (public) | `wrangler.toml` | `POST /ask`, `GET /health` | Turnstile (optional) + CORS allowlist |
+| MCP (internal) | `wrangler.mcp.toml` | `POST /mcp`, `GET /health` | `Authorization: Bearer` (fail closed) |
+| Query (rockenhaus) | `wrangler.rockenhaus.toml` | `POST /ask`, `GET /health` | CORS allowlist (court-record origins) |
 
-Deploy separately so browser traffic and agent traffic can bind different AI Search instances if you want.
+Deploy separately so browser traffic and agent traffic can bind different AI Search instances.
 
 ```sh
-npm run deploy       # query Worker
-npm run deploy:mcp   # MCP Worker
+npm run deploy              # public query Worker
+npm run deploy:mcp          # MCP Worker
+npm run deploy:rockenhaus   # rockenhaus query Worker (uses committed wrangler file)
 wrangler secret put MCP_TOKEN -c wrangler.mcp.toml
 wrangler secret put TURNSTILE_SECRET   # optional; skips verification when unset
 ```
@@ -101,13 +110,114 @@ wrangler secret put TURNSTILE_SECRET   # optional; skips verification when unset
 export R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... CLOUDFLARE_ACCOUNT_ID=...
 export CORPUS_GIT_ORG=your-org GITHUB_TOKEN=...   # for sync-runner clone auth
 
-npm run sync:dry     # plan upload for the default `corpus` target
-npm run sync         # upload + prune
-npm run sync:run     # isolated clone root, sync all targets, optional reindex
+npm run sync:dry          # plan upload for the default `corpus` target
+npm run sync              # upload + prune
+npm run sync:public       # skyphusion public target (when targets.json has it)
+npm run sync:internal     # skyphusion internal target
+npm run sync:rockenhaus   # rockenhaus target
+npm run sync:run          # isolated clone root, sync all targets, optional reindex
 ```
 
+Useful npm scripts (also `npx` CLIs for the first two after install):
+
+| Script | Role |
+| --- | --- |
+| `search-mcp-sync` / `npm run sync` | One-target R2 sync |
+| `search-mcp-sync-run` / `npm run sync:run` | Clone/fetch + multi-target + reindex |
+| `npm run guard:targets` | Additive-only targets.json check |
+| `npm run escrow` | Age-escrow + restore proof for targets secret |
+| `npm run materialize-config` | CI: write wrangler + targets from env secrets |
+
 The sync remaps non-native extensions (`.ts`, `.tsx`, extensionless `Dockerfile`, `.service`, etc.) to `.txt` keys so AI Search indexes them. See `scripts/sync-ingest.mjs`.
 
+### Bounding a corpus: `includePaths` vs `excludePaths`
+
+`excludePaths` is a denylist and is **fail-open**: add a new top-level file to a
+repo and it silently joins the corpus. That is fine for a docs site. It is wrong
+whenever the corpus boundary actually matters, because "we forgot to exclude it"
+becomes a real incident.
+
+`includePaths` / `excludePaths` may sit at the **top level** (same rule for every
+target that lists the repo) or **nested under a target** (search-mcp#62). When
+both set the same repo, the per-target map wins for that target only -- so a
+repo can be full-tree on internal and `_corpus/`-only on a public court surface.
+
+`includePaths` is an allowlist and is **fail-closed**: when a repo has an entry,
+only paths under those prefixes are eligible and everything else is refused.
+
+```json
+{
+  "includePaths": { "my-repo": ["docs/", "_corpus/"] },
+  "excludePaths": { "my-repo": ["docs/internal/"] }
+}
+```
+
+The two compose subtractively: `includePaths` decides what is eligible, then
+`excludePaths` subtracts from that. A denylist entry can never add a path back.
+
+Omit a repo from `includePaths` to keep the previous behaviour. An entry that is
+present but **empty** (`[]`) is a configuration error, not a way to say "index
+nothing": those are different states and collapsing them into the permissive one
+is how a whole repo joins a corpus by accident.
+
+#### A present allowlist must earn its keep
+
+An allowlist that quietly matches nothing is worse than no allowlist. The sync
+plans zero objects, the mirror prune deletes the corpus that was there, the
+reindex succeeds over nothing, and the answer surface returns a confident nothing
+with every status light green. So each of these refuses the run (exit 2) before
+anything uploads or is pruned:
+
+| Refusal | Meaning |
+| --- | --- |
+| `include_paths_no_match` | An entry matched zero git-tracked files (a typo, or a directory that moved) |
+| `include_paths_entry_empty` | The entry is `[]` |
+| `include_paths_entry_invalid` | An entry cannot match a git path (absolute, `..`, backslash, non-string) |
+| `include_paths_entry_not_array` / `include_paths_not_object` | Wrong config shape |
+| `include_paths_unknown_repo` | The entry names a repo no target lists, so nothing reads it |
+| `include_paths_repo_not_cloned` | An allowlisted repo is missing from the clone root |
+| `include_paths_all_excluded` | `excludePaths` removed everything the allowlist selected |
+| `include_paths_all_filtered` | Nothing the allowlist selected survived the ingest filters |
+
+Shape and repo-name checks run for **every** target on every sync, not just the
+one being synced, so a rule that has rotted is caught by the next run of any
+target. A stale `excludePaths` repo name warns rather than refuses: denylist rot
+grows a corpus, allowlist rot empties one.
+
+Verify what a target will actually upload before you trust it:
+
+```sh
+node scripts/sync.mjs my-target --dry-run
+```
+
+### Size cap
+
+Objects over `SYNC_MAX_BYTES` (default 4 MB) are skipped. Skips are summarised at
+the end of the run, not only warned inline, because a file silently missing from
+the corpus looks exactly like a file the corpus does not contain -- the worst
+failure mode for something that answers questions. Pass `--fail-on-skip` to turn
+an incomplete corpus into a failed run.
+
+```sh
+SYNC_MAX_BYTES=$((16 * 1024 * 1024)) node scripts/sync.mjs corpus --fail-on-skip
+```
+
+### Corpus manifest (optional)
+
+A corpus producer can publish a `manifest.json` next to its objects so the widget
+renders citations instead of raw R2 keys. Entries need a `key`; `title`, `url`,
+`page`, and `total_pages` are used when present:
+
+```json
+{ "pages": [
+  { "key": "my-doc/p003.txt", "title": "Deploy guide", "url": "/docs/deploy/", "page": 3, "total_pages": 12 }
+] }
+```
+
+Keys are matched exactly first, then by **suffix**, because the sync namespaces
+every object under its repo name (`<repo>/<path>`) while a producer naturally
+writes its manifest in terms of its own paths.
+
 ### Reindex dispatch
 
 AI Search rejects a new reindex job for two distinct reasons, and `sync-runner` clears both
@@ -118,13 +228,16 @@ before dispatching:
 2. **The post-job cooldown.** Even once a job ends, a new one is refused for a cooldown window
    with `sync_in_cooldown [code: 7020]`. Waiting for the job to end is necessary but not
    sufficient, so we retry until it clears.
+3. **Transient connect failures.** `unable_to_connect_to_ai_search [code: 7017]` on jobs create
+   is retried under the same budget as cooldown (search-mcp#73). Treating it as terminal made
+   the whole sync red while R2 already had the new objects.
 
 Waiting (rather than skipping) means the job we start always lands strictly after our own
 upload, so it sees every object this run wrote. Merge bursts still coalesce: a waiting run holds
 the workflow concurrency group, and GitHub keeps only the newest queued run, so the runs behind
 it collapse instead of each firing their own reindex.
 
-Each wait has its own budget (10 min in-flight, 10 min cooldown) rather than one shared
+Each wait has its own budget (10 min in-flight, 10 min cooldown/connect) rather than one shared
 deadline, since the two are additive on a perfectly healthy path: a run can wait minutes for an
 in-flight reindex and then still owe a cooldown wait.
 
@@ -144,9 +257,30 @@ Copy `public/ask-widget.js` and `public/ask-widget.css` to your docs site:
         data-endpoint="https://search.example.com/ask"
         data-target="#docs-ask"
         data-label="Ask the docs"
+        data-manifest="/corpus-manifest.json"
+        data-empty-text="Nothing in the indexed corpus addresses that."
         data-sitekey="YOUR_TURNSTILE_SITEKEY"></script>
 ```
 
+`data-manifest` is optional; without it sources render as raw object keys. A
+missing or malformed manifest degrades to raw keys rather than breaking answers.
+
+`data-empty-text` is shown when a query returns no retrieved sources, so an
+unsourced answer is never left on screen looking authoritative.
+
+### Per-site system prompts
+
+One deployment can serve several sites. `ORIGIN_PROFILES` is a JSON object in
+`[vars]` mapping an exact request `Origin` to that site's system prompt:
+
+```toml
+ORIGIN_PROFILES = '{"https://docs.example.com":"You are the docs assistant...","https://other.example":"You are..."}'
+```
+
+Precedence is `ORIGIN_PROFILES`, then the legacy blog special-case, then
+`ASSISTANT_SYSTEM_PROMPT`. A malformed value is logged and ignored so a bad var
+cannot take `/ask` down.
+
 ## Who this is for
 
 Operators building documentation search, agent tooling, or internal knowledge bases on [Cloudflare AI Search](https://developers.cloudflare.com/ai-search/) with MCP and a browser widget.
@@ -154,8 +288,9 @@ Operators building documentation search, agent tooling, or internal knowledge ba
 ## Links
 
 - **Deploy guide:** [docs/DEPLOY.md](docs/DEPLOY.md)
-- **Skyphusion Labs:** https://skyphusion.org · **Org:** https://github.com/skyphusion-labs
-- **Related:** [vivijure](https://github.com/skyphusion-labs/vivijure), [postern](https://github.com/skyphusion-labs/postern)
+- **Live ask UIs:** [skyphusion.net/search](https://skyphusion.net/search/) · [vivijure.com](https://vivijure.com) (ask widget)
+- **Skyphusion Labs:** https://skyphusion.org · **Blog:** https://skyphusion.net · **Org:** https://github.com/skyphusion-labs
+- **Related:** [vivijure](https://github.com/skyphusion-labs/vivijure), [postern](https://github.com/skyphusion-labs/postern), [security-audit](https://github.com/skyphusion-labs/security-audit), [prism](https://github.com/skyphusion-labs/prism)
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@skyphusion/search-mcp",
-  "version": "0.2.1",
+  "version": "0.4.0",
   "description": "Bearer-gated MCP server and streaming query Worker over Cloudflare AI Search, plus a git-to-R2 corpus sync toolkit.",
   "license": "AGPL-3.0-only",
   "author": "Conrad Rockenhaus <conrad@skyphusion.org>",
@@ -52,7 +52,7 @@
     "dev:mcp": "wrangler dev -c wrangler.mcp.toml",
     "deploy": "wrangler deploy",
     "deploy:mcp": "wrangler deploy -c wrangler.mcp.toml",
-    "typecheck": "tsc --noEmit",
+    "typecheck": "tsc --noEmit && tsc --noEmit -p tsconfig.scripts.json",
     "test": "vitest run",
     "prepack": "npm run typecheck && npm test",
     "sync": "node scripts/sync.mjs corpus",
@@ -60,13 +60,18 @@
     "sync:internal": "node scripts/sync.mjs internal",
     "sync:dry": "node scripts/sync.mjs corpus --dry-run",
     "sync:run": "node scripts/sync-runner.mjs",
-    "materialize-config": "node scripts/materialize-config.mjs"
+    "materialize-config": "node scripts/materialize-config.mjs",
+    "escrow": "node scripts/escrow-targets.mjs",
+    "guard:targets": "node scripts/guard-targets-additive.mjs",
+    "deploy:rockenhaus": "wrangler deploy -c wrangler.rockenhaus.toml",
+    "sync:rockenhaus": "node scripts/sync.mjs rockenhaus"
   },
   "dependencies": {
     "@aws-sdk/client-s3": "^3.888.0"
   },
   "devDependencies": {
     "@cloudflare/workers-types": "^5.20260713.1",
+    "@types/node": "^22.15.0",
     "@vitest/coverage-istanbul": "^4.1.10",
     "typescript": "^7.0.2",
     "vitest": "^4.1.10",

package/public/ask-widget.js

@@ -16,6 +16,12 @@
   var label = script.getAttribute("data-label") || "Ask";
   var placeholder = script.getAttribute("data-placeholder") || "Ask a question about the docs";
   var sitekey = script.getAttribute("data-sitekey") || "";
+  // Optional corpus manifest: maps an R2 object key to a human title, a URL,
+  // and a page number, so sources render as citations instead of raw keys.
+  var manifestUrl = script.getAttribute("data-manifest") || "";
+  var emptyText =
+    script.getAttribute("data-empty-text") ||
+    "Nothing in the indexed corpus addresses that.";
 
   function ready(fn) {
     if (document.readyState !== "loading") fn();
@@ -69,6 +75,97 @@
       }
     }
 
+
+    // ---- corpus manifest -------------------------------------------------
+    //
+    // Without a manifest the widget can only show the R2 object key, which is
+    // an implementation detail ("repo/docs/DEPLOY.md"). With one it can show
+    // what a reader actually needs: the document title, a link, and a page.
+    //
+    // Entries are matched by exact key first, then by SUFFIX. Suffix matching
+    // matters because the sync prefixes every object with its repo name, so the
+    // key in R2 is "<repo>/<path>" while a corpus producer naturally writes its
+    // manifest in terms of its own paths. Suffix matching lets the producer stay
+    // ignorant of how the sync namespaces things.
+    var manifest = null;
+    var manifestIndex = null;
+
+    function indexManifest(data) {
+      var entries = [];
+      if (Array.isArray(data)) entries = data;
+      else if (data && Array.isArray(data.pages)) entries = data.pages;
+      else if (data && Array.isArray(data.objects)) entries = data.objects;
+      var byKey = {};
+      entries.forEach(function (e) {
+        if (e && typeof e.key === "string") byKey[e.key] = e;
+      });
+      return { byKey: byKey, entries: entries };
+    }
+
+    function loadManifest() {
+      if (!manifestUrl) return Promise.resolve(null);
+      if (manifest) return Promise.resolve(manifest);
+      return fetch(manifestUrl)
+        .then(function (r) {
+          return r.ok ? r.json() : null;
+        })
+        .then(function (data) {
+          if (!data) return null;
+          manifest = data;
+          manifestIndex = indexManifest(data);
+          return data;
+        })
+        .catch(function () {
+          // A missing or broken manifest must not break answers. Degrade to
+          // showing raw keys rather than failing the request.
+          return null;
+        });
+    }
+
+    function lookup(key) {
+      if (!manifestIndex) return null;
+      if (manifestIndex.byKey[key]) return manifestIndex.byKey[key];
+      for (var i = 0; i < manifestIndex.entries.length; i++) {
+        var e = manifestIndex.entries[i];
+        if (e && typeof e.key === "string" && key.indexOf(e.key) >= 0) {
+          var at = key.length - e.key.length;
+          if (at >= 0 && key.slice(at) === e.key) return e;
+        }
+      }
+      return null;
+    }
+
+    function citationNode(key) {
+      var entry = lookup(key);
+      var li = document.createElement("li");
+      if (!entry) {
+        li.textContent = key;
+        return li;
+      }
+      var label = entry.title || key;
+      if (entry.url) {
+        var a = document.createElement("a");
+        a.setAttribute("href", entry.url);
+        a.textContent = label;
+        li.appendChild(a);
+      } else {
+        li.appendChild(document.createTextNode(label));
+      }
+      var bits = [];
+      if (entry.page) {
+        bits.push("page " + entry.page + (entry.total_pages ? " of " + entry.total_pages : ""));
+      }
+      if (entry.case_number) bits.push("case " + entry.case_number);
+      if (entry.filed_date) bits.push("filed " + entry.filed_date);
+      if (bits.length) {
+        var span = document.createElement("span");
+        span.className = "vjask-source-meta";
+        span.textContent = " (" + bits.join(", ") + ")";
+        li.appendChild(span);
+      }
+      return li;
+    }
+
     function renderSources(chunks) {
       var seen = {};
       var items = [];
@@ -80,12 +177,18 @@
         }
       });
       if (!items.length) return;
-      sources.innerHTML = "<li class='vjask-sources-title'>Sources</li>" +
-        items.map(function (k) {
-          return "<li>" + k.replace(/[&<>]/g, function (ch) {
-            return { "&": "&amp;", "<": "&lt;", ">": "&gt;" }[ch];
-          }) + "</li>";
-        }).join("");
+      sawSources = true;
+      // Built via the DOM API rather than innerHTML: entries come from the
+      // corpus manifest, and concatenating them into markup would be an
+      // injection path through a data file.
+      sources.textContent = "";
+      var title = document.createElement("li");
+      title.className = "vjask-sources-title";
+      title.textContent = "Sources";
+      sources.appendChild(title);
+      items.forEach(function (k) {
+        sources.appendChild(citationNode(k));
+      });
       sources.hidden = false;
     }
 
@@ -109,6 +212,8 @@
       if (delta && delta.content) answer.textContent += delta.content;
     }
 
+    var sawSources = false;
+
     form.addEventListener("submit", function (ev) {
       ev.preventDefault();
       var q = input.value.trim();
@@ -116,8 +221,10 @@
       btn.disabled = true;
       answer.textContent = "";
       sources.hidden = true;
-      sources.innerHTML = "";
+      sources.textContent = "";
+      sawSources = false;
 
+      loadManifest().then(function () {
       fetch(endpoint, {
         method: "POST",
         headers: { "content-type": "application/json" },
@@ -153,6 +260,12 @@
           answer.textContent = "Sorry, something went wrong (" + err.message + ").";
         })
         .finally(function () {
+          // No retrieved sources means nothing in the corpus matched. Say that
+          // outright rather than leaving a plausible-looking unsourced answer
+          // on screen.
+          if (!sawSources && !answer.textContent.trim()) {
+            answer.textContent = emptyText;
+          }
           btn.disabled = false;
           if (sitekey && window.turnstile) {
             try {
@@ -160,6 +273,7 @@
             } catch (e) {}
           }
         });
+      });
     });
   });
 })();

package/scripts/sync-ingest.mjs

@@ -123,19 +123,323 @@ export function ingestKind(relPath) {
   return DOC_EXT.has(ext) || basename(relPath).startsWith("README") ? "doc" : "code";
 }
 
+
+/**
+ * One prefix rule, shared by the allowlist and the denylist so the two can
+ * never drift. An entry is a repo-relative path: with a trailing "/" it means
+ * the subtree, without one it means that exact file or the subtree rooted at
+ * that name (so "docs" matches "docs" and "docs/a.md" but never "docsite/a.md").
+ */
+export function matchesPathPrefix(relPath, prefix) {
+  if (typeof prefix !== "string" || prefix === "") return false;
+  if (prefix.endsWith("/")) return relPath.startsWith(prefix);
+  return relPath === prefix || relPath.startsWith(prefix + "/");
+}
+
 /**
- * Per-repo corpus exclusion (targets.json `excludePaths`). An entry is a repo-relative
- * path: with a trailing "/" it excludes the whole subtree; without, it excludes that
- * exact file or the subtree rooted at that name.
+ * Per-repo corpus exclusion (targets.json `excludePaths`). Denylist, applied on
+ * top of the allowlist. Absent or empty means "exclude nothing".
  */
 export function isExcludedPath(relPath, prefixes) {
   for (const p of prefixes || []) {
-    if (!p) continue;
-    if (p.endsWith("/")) {
-      if (relPath.startsWith(p)) return true;
-    } else if (relPath === p || relPath.startsWith(p + "/")) {
-      return true;
-    }
+    if (matchesPathPrefix(relPath, p)) return true;
   }
   return false;
 }
+
+/**
+ * Per-repo corpus ALLOWLIST (targets.json `includePaths`). When a repo has an
+ * entry, ONLY paths under one of its prefixes are eligible for the corpus;
+ * everything else is refused. When a repo has NO entry (undefined/null) every
+ * path is eligible and `excludePaths` alone applies, so a config that never
+ * mentions includePaths behaves exactly as it did before.
+ *
+ * Why both exist. A denylist is fail-OPEN: add a new top-level file to a repo
+ * and it silently joins the corpus. That is fine for a docs site and wrong for
+ * a corpus whose boundary matters -- a court-record mirror, anything with a
+ * privileged directory, anything where "we forgot to exclude it" is an incident
+ * rather than noise. An allowlist is fail-CLOSED: a new path is ineligible
+ * until someone says otherwise.
+ *
+ * An entry that is PRESENT but empty (or malformed) matches nothing rather than
+ * everything. "The operator wrote an allowlist" and "the operator wrote no
+ * allowlist" are different states and must not collapse into the permissive
+ * one; the config validation below turns that state into a hard error before a
+ * sync ever reaches this matcher.
+ */
+export function isIncludedPath(relPath, prefixes) {
+  if (prefixes === undefined || prefixes === null) return true;
+  if (!Array.isArray(prefixes)) return false;
+  for (const p of prefixes) {
+    if (matchesPathPrefix(relPath, p)) return true;
+  }
+  return false;
+}
+
+/**
+ * Thrown for every includePaths refusal. `code` names the specific state so a
+ * test can prove a failure happened for the stated reason and not by accident.
+ */
+export class IncludePathsError extends Error {
+  constructor(code, message) {
+    super(message);
+    this.name = "IncludePathsError";
+    this.code = code;
+  }
+}
+
+function describeType(v) {
+  if (v === null) return "null";
+  if (Array.isArray(v)) return "array";
+  return typeof v;
+}
+
+function q(v) {
+  return typeof v === "string" ? JSON.stringify(v) : describeType(v) + " " + JSON.stringify(v);
+}
+
+/** Every repo named by any target. An includePaths key outside this set reaches nothing. */
+export function knownTargetRepos(cfg) {
+  const repos = new Set();
+  for (const target of Object.values(cfg?.targets || {})) {
+    for (const repo of target?.repos || []) repos.add(repo);
+  }
+  return repos;
+}
+
+/**
+ * Resolve include/exclude path maps for ONE target (search-mcp#62).
+ *
+ * Precedence: per-target maps win per-repo over top-level maps. Top-level alone
+ * keeps the pre-#62 shape (the same rule for every target that lists the repo).
+ * A repo listed only under another target's nested map is not visible here.
+ *
+ * Why both layers: the original top-level key was fine while every exclusion
+ * applied to exactly one target; the moment a repo is deliberately indexed at
+ * two granularities (public vs internal), a single map cannot express it.
+ */
+export function pathMapsForTarget(cfg, targetName) {
+  const target = cfg?.targets?.[targetName] || {};
+  const topInclude =
+    cfg?.includePaths && typeof cfg.includePaths === "object" && !Array.isArray(cfg.includePaths)
+      ? cfg.includePaths
+      : {};
+  const topExclude =
+    cfg?.excludePaths && typeof cfg.excludePaths === "object" && !Array.isArray(cfg.excludePaths)
+      ? cfg.excludePaths
+      : {};
+  const tInclude =
+    target.includePaths && typeof target.includePaths === "object" && !Array.isArray(target.includePaths)
+      ? target.includePaths
+      : {};
+  const tExclude =
+    target.excludePaths && typeof target.excludePaths === "object" && !Array.isArray(target.excludePaths)
+      ? target.excludePaths
+      : {};
+  return {
+    includePaths: { ...topInclude, ...tInclude },
+    excludePaths: { ...topExclude, ...tExclude },
+  };
+}
+
+function isPathMapObject(v) {
+  return v !== undefined && v !== null && typeof v === "object" && !Array.isArray(v);
+}
+
+/** Validate one path-map object against a known-repo set. Returns error objects. */
+function validateIncludePathsMap(includePaths, known, label) {
+  const errors = [];
+  if (includePaths === undefined || includePaths === null) return errors;
+  if (!isPathMapObject(includePaths)) {
+    errors.push({
+      code: "include_paths_not_object",
+      message:
+        `${label} must be an object keyed by repo, got ${describeType(includePaths)}. ` +
+        `Shape: "includePaths": { "my-repo": ["_corpus/"] }.`,
+    });
+    return errors;
+  }
+  for (const [repo, prefixes] of Object.entries(includePaths)) {
+    if (repo.startsWith("_")) continue; // documentation keys in the example file
+    if (!known.has(repo)) {
+      errors.push({
+        code: "include_paths_unknown_repo",
+        message:
+          `${label} names repo ${q(repo)}, which is not in the applicable target repo list. ` +
+          `Nothing reads that rule, and a rule nothing reads is indistinguishable from a rule ` +
+          `that works. Add the repo to the target or delete the entry. Applicable repos: ` +
+          `${[...known].join(", ") || "(none)"}.`,
+      });
+    }
+    errors.push(...validateIncludePathsEntry(repo, prefixes));
+  }
+  return errors;
+}
+
+/** Shape check for one repo entry. Returns {code, message} objects, never throws. */
+export function validateIncludePathsEntry(repo, prefixes) {
+  const errors = [];
+  if (!Array.isArray(prefixes)) {
+    errors.push({
+      code: "include_paths_entry_not_array",
+      message:
+        `includePaths for repo ${q(repo)} must be an array of repo-relative path prefixes, ` +
+        `got ${describeType(prefixes)}. Shape: "includePaths": { "my-repo": ["_corpus/"] }.`,
+    });
+    return errors;
+  }
+  if (prefixes.length === 0) {
+    errors.push({
+      code: "include_paths_entry_empty",
+      message:
+        `includePaths for repo ${q(repo)} is an empty array, which is a configuration error, ` +
+        `not an instruction to index nothing. Remove the repo entry to index all of it, or ` +
+        `list the prefixes to index.`,
+    });
+    return errors;
+  }
+  for (const p of prefixes) {
+    if (typeof p !== "string" || p.trim() === "") {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: entries must be non-empty strings.`,
+      });
+    } else if (p.startsWith("/")) {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: entries are repo-relative and ` +
+          `must not start with "/". A git path never does, so this would match nothing.`,
+      });
+    } else if (p.split("/").includes("..")) {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: ".." is not something a git path ` +
+          `can start with, so this would match nothing.`,
+      });
+    } else if (p.includes("\\")) {
+      errors.push({
+        code: "include_paths_entry_invalid",
+        message:
+          `includePaths for repo ${q(repo)} has entry ${q(p)}: git paths use "/" as the ` +
+          `separator, so a backslash would match nothing.`,
+      });
+    }
+  }
+  return errors;
+}
+
+/**
+ * Whole-config check for targets.json `includePaths` (top-level AND per-target,
+ * search-mcp#62). Returns {code, message} objects, never throws, so a caller can
+ * report all of them at once.
+ *
+ * Rejects an entry naming a repo no applicable target lists: a rule nothing
+ * reads is indistinguishable from a rule that works, which is how an allowlist
+ * rots. Per-target maps are checked against that target's repos only.
+ */
+export function validateIncludePathsConfig(cfg) {
+  const errors = [];
+  const knownAll = knownTargetRepos(cfg);
+  errors.push(...validateIncludePathsMap(cfg?.includePaths, knownAll, "targets.json includePaths"));
+  for (const [name, target] of Object.entries(cfg?.targets || {})) {
+    if (!target || typeof target !== "object") continue;
+    const known = new Set(Array.isArray(target.repos) ? target.repos : []);
+    errors.push(
+      ...validateIncludePathsMap(target.includePaths, known, `targets.${name}.includePaths`),
+    );
+  }
+  return errors;
+}
+
+/** Fail-closed wrapper: throws IncludePathsError carrying the first code and every message. */
+export function assertIncludePathsConfig(cfg) {
+  const errors = validateIncludePathsConfig(cfg);
+  if (!errors.length) return;
+  throw new IncludePathsError(errors[0].code, errors.map((e) => e.message).join(" "));
+}
+
+/**
+ * excludePaths keys (top-level OR nested under a target) naming a repo no
+ * applicable target lists. Denylist rot is quieter than allowlist rot (the
+ * corpus grows rather than empties), so this is a warning at the call site,
+ * not a refusal. Returns unique repo names.
+ */
+export function unknownExcludePathsRepos(cfg) {
+  const out = new Set();
+  const knownAll = knownTargetRepos(cfg);
+  const top = cfg?.excludePaths;
+  if (isPathMapObject(top)) {
+    for (const repo of Object.keys(top)) {
+      if (!repo.startsWith("_") && !knownAll.has(repo)) out.add(repo);
+    }
+  }
+  for (const target of Object.values(cfg?.targets || {})) {
+    if (!target || typeof target !== "object") continue;
+    const nested = target.excludePaths;
+    if (!isPathMapObject(nested)) continue;
+    const known = new Set(Array.isArray(target.repos) ? target.repos : []);
+    for (const repo of Object.keys(nested)) {
+      if (!repo.startsWith("_") && !known.has(repo)) out.add(repo);
+    }
+  }
+  return [...out];
+}
+
+/**
+ * Pure path selection for one repo: allowlist first, then denylist on top, so
+ * the two compose as defense in depth rather than one replacing the other.
+ *
+ * `relPaths` is the repo git-tracked file list. Refuses, loudly:
+ *
+ *   include_paths_entry_*       the entry is malformed or empty
+ *   include_paths_no_match      an entry matched ZERO tracked files
+ *   include_paths_all_excluded  the allowlist matched, excludePaths took it all
+ *
+ * The zero-match refusal is the point of the whole mechanism. A prefix that
+ * matches nothing (a rename, a misspelling, a directory that moved) otherwise
+ * plans an empty corpus, the mirror prune deletes what was there, the reindex
+ * succeeds over nothing, and the answer surface returns a confident nothing with
+ * every status light green. Coverage is measured against the tracked list before
+ * exclusion, so "this prefix names nothing in the repo" stays a distinct
+ * diagnostic from "everything it named was excluded".
+ */
+export function selectRepoPaths(repo, relPaths, opts = {}) {
+  const { includePrefixes, excludePrefixes } = opts;
+  const paths = relPaths || [];
+  if (includePrefixes === undefined || includePrefixes === null) {
+    return paths.filter((rel) => !isExcludedPath(rel, excludePrefixes));
+  }
+
+  const shapeErrors = validateIncludePathsEntry(repo, includePrefixes);
+  if (shapeErrors.length) {
+    throw new IncludePathsError(shapeErrors[0].code, shapeErrors.map((e) => e.message).join(" "));
+  }
+
+  const unmatched = includePrefixes.filter((p) => !paths.some((rel) => matchesPathPrefix(rel, p)));
+  if (unmatched.length) {
+    throw new IncludePathsError(
+      "include_paths_no_match",
+      `includePaths for repo ${q(repo)} matched 0 git-tracked files: ` +
+        `${unmatched.map(q).join(", ")} (the repo has ${paths.length} tracked file(s)). ` +
+        `An allowlist entry that matches nothing would sync an empty corpus over a healthy one ` +
+        `and report success, so the sync refuses instead. Fix the prefix in targets.json ` +
+        `(repo-relative, trailing slash for a subtree), or drop the entry if the path is gone. ` +
+        `Only the prefixes named above matched nothing.`,
+    );
+  }
+
+  const included = paths.filter((rel) => isIncludedPath(rel, includePrefixes));
+  const kept = included.filter((rel) => !isExcludedPath(rel, excludePrefixes));
+  if (kept.length === 0) {
+    throw new IncludePathsError(
+      "include_paths_all_excluded",
+      `includePaths for repo ${q(repo)} selected ${included.length} file(s) and excludePaths ` +
+        `removed every one of them. That leaves nothing to sync for this repo, which is an ` +
+        `empty corpus dressed as a successful run. Widen includePaths or narrow excludePaths.`,
+    );
+  }
+  return kept;
+}

package/scripts/sync-runner.mjs

@@ -172,10 +172,23 @@ export const REINDEX_POLL_MS = 15 * 1000;
  * AI Search refuses a new job for a cooldown window after the previous one ENDS, distinct from
  * the job being in flight. Measured 2026-07-16: rejected 10s after a job ended, accepted at
  * 32s. Waiting for `ended_at` alone is necessary but not sufficient.
+ *
+ * Also treated as transient (2026-08-04 live corpus-sync): `unable_to_connect_to_ai_search
+ * [code: 7017]` on jobs create for skyphusion-public -- the instance was healthy (list jobs
+ * worked; a later create succeeded). Retrying under the same budget as cooldown is correct;
+ * treating 7017 as terminal made the whole sync red while R2 already had the new objects.
  */
-export function isCooldownError(err) {
+export function isTransientReindexError(err) {
   const text = `${err?.stdout || ""}${err?.stderr || ""}${err?.message || ""}`;
-  return /sync_in_cooldown|code:\s*7020/.test(text);
+  return (
+    /sync_in_cooldown|code:\s*7020/.test(text) ||
+    /unable_to_connect_to_ai_search|code:\s*7017/.test(text)
+  );
+}
+
+/** @deprecated prefer isTransientReindexError; kept so older call sites and tests keep working. */
+export function isCooldownError(err) {
+  return isTransientReindexError(err);
 }
 
 /**
@@ -218,13 +231,14 @@ export async function awaitReindexSlot(instance, deps) {
 }
 
 /**
- * Dispatch a reindex, retrying while AI Search reports its post-job cooldown.
+ * Dispatch a reindex, retrying while AI Search reports a TRANSIENT refuse
+ * (post-job cooldown 7020, or unable_to_connect_to_ai_search 7017).
  *
- * Cooldown is transient and short (measured well under the budget), so the routine burst case
- * clears here and never goes red. Exhausting the budget therefore means something genuinely
- * anomalous upstream, and we fail loud rather than exit green: a green run while indexing is
- * actually stalled is the work-blind failure mode this whole issue exists to remove. The
- * message carries the honest blast radius so the red is actionable, not alarming.
+ * These clears are short relative to the budget, so the routine burst case clears here and
+ * never goes red. Exhausting the budget means something genuinely anomalous upstream, and we
+ * fail loud rather than exit green: a green run while indexing is actually stalled is the
+ * work-blind failure mode this whole issue exists to remove. The message carries the honest
+ * blast radius so the red is actionable, not alarming.
  */
 export async function dispatchWithCooldownRetry(instance, deps) {
   const {
@@ -242,17 +256,21 @@ export async function dispatchWithCooldownRetry(instance, deps) {
       runReindexOnce(instance);
       return { retried: announced };
     } catch (err) {
-      if (!isCooldownError(err)) throw err;
+      if (!isTransientReindexError(err)) throw err;
       if (now() >= deadline) {
         throw new Error(
-          `reindex for ${instance} blocked by AI Search cooldown for longer than ` +
-            `${Math.round(timeoutMs / 1000)}s. The R2 corpus uploaded OK and no data is lost; ` +
-            "the index lags until the next sync or the daily backstop. A cooldown this " +
-            "persistent is upstream and anomalous, so this run fails loudly on purpose.",
+          `reindex for ${instance} blocked by transient AI Search errors for longer than ` +
+            `${Math.round(timeoutMs / 1000)}s (cooldown 7020 and/or connect 7017). ` +
+            "The R2 corpus uploaded OK and no data is lost; the index lags until the next " +
+            "sync or the daily backstop. Persistence this long is upstream and anomalous, " +
+            "so this run fails loudly on purpose.",
         );
       }
       if (!announced) {
-        log(`  ${instance} is in AI Search cooldown; retrying until it clears.`);
+        log(
+          `  ${instance} hit a transient AI Search refuse (cooldown or unable_to_connect); ` +
+            "retrying until it clears.",
+        );
         announced = true;
       }
       await sleep(pollMs);

package/scripts/sync.mjs

@@ -36,7 +36,11 @@ import {
   ingestContentType,
   ingestKind,
   fileExt,
-  isExcludedPath,
+  selectRepoPaths,
+  assertIncludePathsConfig,
+  unknownExcludePathsRepos,
+  pathMapsForTarget,
+  IncludePathsError,
 } from "./sync-ingest.mjs";
 import {
   resolveTargetsPath,
@@ -47,11 +51,18 @@ import {
 const HERE = dirname(fileURLToPath(import.meta.url));
 const TARGETS_PATH = resolveTargetsPath(HERE);
 const REPO_ROOT = defaultRepoRoot(HERE, TARGETS_PATH);
-const MAX_BYTES = 4 * 1024 * 1024;
+// Objects larger than this are skipped. Configurable because the right value
+// depends on the corpus: 4 MB is fine for a source tree and wrong for scanned
+// PDFs, where the single largest document is often the one that matters most.
+// Skips are also summarised at the end of the run, not only warned inline: a
+// warning in a long CI log is a silent omission in practice.
+const MAX_BYTES = Number(process.env.SYNC_MAX_BYTES || 4 * 1024 * 1024);
 
 const args = process.argv.slice(2);
 const DRY = args.includes("--dry-run");
 const SKIP_GITHUB = args.includes("--no-github-verify") || process.env.SYNC_SKIP_GITHUB_VERIFY === "1";
+// Opt-in strictness: turn an incomplete corpus into a failed run.
+const FAIL_ON_SKIP = args.includes("--fail-on-skip");
 const targetName = args.find((a) => !a.startsWith("--"));
 
 const SKIP_EXT = new Set([
@@ -123,16 +134,33 @@ function loadConfig() {
   return JSON.parse(readFileSync(TARGETS_PATH, "utf8"));
 }
 
-function planRepo(repo, excludePrefixes = []) {
+function planRepo(repo, excludePrefixes, includePrefixes, skipped = []) {
   const repoDir = join(REPO_ROOT, repo);
+  const hasAllowlist = includePrefixes !== undefined && includePrefixes !== null;
   if (!existsSync(repoDir)) {
+    // A repo with an allowlist has been described exactly: it contributes these
+    // paths and nothing else. A missing clone contributes nothing, which is the
+    // same empty-corpus-with-green-lights failure the allowlist exists to stop.
+    if (hasAllowlist) {
+      throw new IncludePathsError(
+        "include_paths_repo_not_cloned",
+        `includePaths for repo "${repo}" cannot be enforced: not cloned at ${repoDir}. ` +
+          `A repo with an allowlist is expected to contribute exactly those paths, so a ` +
+          `missing clone refuses the sync instead of quietly syncing nothing for it.`,
+      );
+    }
     console.warn(`  ! skip ${repo}: not cloned at ${repoDir}`);
     return [];
   }
+  // Allowlist first, then denylist on top; both refusals happen here, before a
+  // single object is uploaded or pruned.
+  const candidates = selectRepoPaths(repo, trackedFiles(repoDir), {
+    includePrefixes,
+    excludePrefixes,
+  });
   const items = [];
-  for (const rel of trackedFiles(repoDir)) {
+  for (const rel of candidates) {
     if (shouldSkip(rel)) continue;
-    if (isExcludedPath(rel, excludePrefixes)) continue;
     const abs = join(repoDir, rel);
     let size;
     try {
@@ -141,7 +169,10 @@ function planRepo(repo, excludePrefixes = []) {
       continue;
     }
     if (size > MAX_BYTES) {
-      console.warn(`  ! skip ${repo}/${rel}: ${(size / 1048576).toFixed(1)} MB over 4 MB`);
+      const mb = (size / 1048576).toFixed(1);
+      const capMb = (MAX_BYTES / 1048576).toFixed(1);
+      console.warn(`  ! skip ${repo}/${rel}: ${mb} MB over the ${capMb} MB cap`);
+      skipped.push({ repo, path: rel, mb });
       continue;
     }
     const sample = readTextSample(abs, size);
@@ -161,12 +192,23 @@ function planRepo(repo, excludePrefixes = []) {
       },
     });
   }
+  if (hasAllowlist && items.length === 0) {
+    throw new IncludePathsError(
+      "include_paths_all_filtered",
+      `includePaths for repo "${repo}" selected ${candidates.length} file(s) and none of them ` +
+        `survived the ingest filters (build/vendor directories, lockfiles, .env, binaries, the ` +
+        `size cap, or the text sniff). Nothing would upload for this repo, so the sync refuses ` +
+        `rather than publish an empty corpus.`,
+    );
+  }
   return items;
 }
 
 async function main() {
   if (!targetName) {
-    console.error("usage: node scripts/sync.mjs <target> [--dry-run] [--no-github-verify]");
+    console.error(
+      "usage: node scripts/sync.mjs <target> [--dry-run] [--no-github-verify] [--fail-on-skip]",
+    );
     process.exit(2);
   }
   const cfg = loadConfig();
@@ -176,6 +218,17 @@ async function main() {
     process.exit(2);
   }
 
+  // Config-level allowlist checks run for every target, not just the one being
+  // synced: a rule that names a repo no target lists is rot, and rot found only
+  // when someone syncs that one target is rot found late.
+  assertIncludePathsConfig(cfg);
+  for (const repo of unknownExcludePathsRepos(cfg)) {
+    console.warn(
+      `  ! excludePaths names repo '${repo}', which no target lists in its repos. ` +
+        "That rule is inert; delete it or add the repo to a target.",
+    );
+  }
+
   assertPublicCorpusBoundary(cfg, targetName);
   if (targetName === "public" && !SKIP_GITHUB) {
     const gh = await assertPublicGithubVisibility(cfg);
@@ -194,14 +247,39 @@ async function main() {
   console.log(`Target '${targetName}' -> bucket ${target.bucket} (instance ${target.instance})`);
   console.log(`Repo root: ${REPO_ROOT}${DRY ? "  [DRY RUN]" : ""}`);
 
+  // #62: per-target path maps win over top-level for this target.
+  const pathMaps = pathMapsForTarget(cfg, targetName);
+
   const plan = [];
+  const skipped = [];
   for (const repo of target.repos) {
-    const items = planRepo(repo, cfg.excludePaths?.[repo]);
+    const items = planRepo(
+      repo,
+      pathMaps.excludePaths?.[repo],
+      pathMaps.includePaths?.[repo],
+      skipped,
+    );
     if (items.length) console.log(`  + ${repo}: ${items.length} files`);
     plan.push(...items);
   }
   console.log(`Planned ${plan.length} objects for ${target.bucket}.`);
 
+  // Summarise oversize skips. A file silently missing from the corpus looks
+  // exactly like a file the corpus does not contain, which is the worst
+  // possible failure mode for something that answers questions.
+  if (skipped.length) {
+    console.warn(`\n  ! ${skipped.length} file(s) skipped for exceeding the size cap:`);
+    for (const s of skipped) console.warn(`      ${s.repo}/${s.path} (${s.mb} MB)`);
+    console.warn(
+      "    These are NOT in the corpus. Raise SYNC_MAX_BYTES, extract their text into\n" +
+        "    a smaller form, or accept that nothing can answer from them.",
+    );
+    if (FAIL_ON_SKIP) {
+      console.error("--fail-on-skip: refusing to continue with an incomplete corpus.");
+      process.exit(1);
+    }
+  }
+
   if (DRY) {
     for (const it of plan.slice(0, 25)) console.log(`    ${it.key}`);
     if (plan.length > 25) console.log(`    ... and ${plan.length - 25} more`);
@@ -283,6 +361,12 @@ async function main() {
 }
 
 main().catch((err) => {
+  // A config refusal is an operator error, not a crash: print the diagnostic
+  // without a stack, and exit 2 like the other targets.json failures.
+  if (err instanceof IncludePathsError) {
+    console.error(`\n${err.message}\n  [${err.code}]`);
+    process.exit(2);
+  }
   console.error(err);
   process.exit(1);
 });

package/scripts/targets.json.example

@@ -1,5 +1,11 @@
 {
   "_comment": "Copy to targets.json (./targets.json for npm CLI, or scripts/targets.json in a git clone). repos are directory names under SYNC_REPO_ROOT (default: cwd for npm, parent of this repo for dev clone). Only git-tracked files are uploaded.",
+  "_includePaths_comment": "ALLOWLIST, optional, per repo. When present, ONLY paths under these prefixes are eligible for the corpus and everything else is refused. Use it when the corpus boundary matters: a denylist is fail-OPEN, so a new top-level file silently joins the corpus, whereas an allowlist is fail-CLOSED. Omit a repo to keep the old behaviour of admitting everything; an entry that is present but EMPTY is an error, not a way to index nothing. May live at the top level (applies to every target that lists the repo) OR nested under a target (search-mcp#62); per-target wins per repo when both set the same key.",
+  "_includePaths_refusals": "A present allowlist must earn its keep: an entry matching zero git-tracked files, an entry naming a repo no applicable target lists, a repo whose clone is missing, or a selection that excludePaths or the ingest filters empty out all refuse the sync (exit 2). An allowlist that quietly matches nothing would prune a healthy corpus and reindex over the emptiness with every status light green.",
+  "includePaths": {
+    "example-repo": ["docs/"]
+  },
+  "_excludePaths_comment": "DENYLIST, optional, per repo. Applied after includePaths. Same top-level vs per-target nesting as includePaths.",
   "excludePaths": {
     "example-repo": ["vendor/private-notes/"]
   },
@@ -9,6 +15,15 @@
       "instance": "my-ai-search-instance",
       "bucket": "my-search-corpus",
       "repos": ["example-repo"]
+    },
+    "_per_target_paths_example": {
+      "_comment": "Optional nested includePaths/excludePaths apply only to this target (search-mcp#62). Use when the same repo is indexed at different granularities across targets.",
+      "instance": "other-instance",
+      "bucket": "other-bucket",
+      "repos": ["example-repo"],
+      "includePaths": {
+        "example-repo": ["_corpus/"]
+      }
     }
   }
 }