@skyphusion/search-mcp 0.1.2

package/README.md

@@ -0,0 +1,144 @@
+# search-mcp
+
+[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](LICENSE)
+[![Typecheck](https://github.com/skyphusion-labs/search-mcp/actions/workflows/typecheck.yml/badge.svg)](https://github.com/skyphusion-labs/search-mcp/actions/workflows/typecheck.yml)
+[![CI](https://github.com/skyphusion-labs/search-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/skyphusion-labs/search-mcp/actions/workflows/ci.yml)
+
+Open-source toolkit for [Cloudflare AI Search](https://developers.cloudflare.com/ai-search/):
+
+1. **MCP Worker** (`src/mcp.ts`) -- bearer-gated Streamable-HTTP MCP with a `search` tool for agents.
+2. **Query Worker** (`src/index.ts`) -- CORS + Turnstile + rate-limited `POST /ask` that streams answers for a browser widget.
+3. **Corpus sync** (`scripts/sync.mjs`, `scripts/sync-runner.mjs`) -- git-tracked sources to R2, with extension remapping so TypeScript, Dockerfiles, and other text AI Search would otherwise skip get indexed.
+
+```
+git repos  ->  sync.mjs  ->  R2 bucket  ->  AI Search instance  ->  /ask + /mcp
+```
+
+## Install (npm)
+
+The corpus sync CLIs and ask-widget assets ship on npm as **`@skyphusion/search-mcp`** (the unscoped name `search-mcp` is taken by another project).
+
+```sh
+npm install @skyphusion/search-mcp
+# or run without installing:
+npx --package=@skyphusion/search-mcp search-mcp-sync corpus --dry-run
+```
+
+| Command | Role |
+| --- | --- |
+| `search-mcp-sync` | Upload git-tracked corpus files to R2 for one target |
+| `search-mcp-sync-run` | Clone/fetch repos, sync all targets, optional reindex |
+
+Put `targets.json` in your project root (copy from `node_modules/@skyphusion/search-mcp/scripts/targets.json.example`) or set `SEARCH_MCP_TARGETS`. Clone roots default to the current working directory; override with `SYNC_REPO_ROOT`.
+
+Widget assets after install:
+
+```sh
+cp node_modules/@skyphusion/search-mcp/public/ask-widget.{js,css} ./docs/
+```
+
+Workers (`src/`) deploy from a git clone; see [docs/DEPLOY.md](docs/DEPLOY.md).
+
+## Quick start (from source)
+
+```sh
+npm install
+cp wrangler.toml.example wrangler.toml
+cp wrangler.mcp.toml.example wrangler.mcp.toml
+cp scripts/targets.json.example scripts/targets.json
+# edit the three files for your account, instance, bucket, and repos
+
+npm run typecheck
+npm test
+```
+
+Provision R2 + AI Search, sync your corpus, deploy both Workers. Step-by-step: [docs/DEPLOY.md](docs/DEPLOY.md).
+
+## Skyphusion production
+
+This repo is the production home for Skyphusion AI Search (`search.vivijure.com`,
+`search-internal.vivijure.com`). Config is materialized from GitHub Actions secrets at deploy/sync
+time; do not commit `wrangler.toml`, `wrangler.mcp.toml`, or `scripts/targets.json`.
+
+- [Operator runbook](docs/skyphusion/OPERATOR.md) -- secrets, topology, bootstrap
+- [Cutover record (2026-07-08)](docs/skyphusion/CUTOVER.md) -- migration history, failure log, archive steps
+
+## Workers
+
+| Worker | Entry | Endpoint | Auth |
+| --- | --- | --- | --- |
+| Query | `wrangler.toml` | `POST /ask`, `GET /health` | Turnstile (optional) + CORS allowlist |
+| MCP | `wrangler.mcp.toml` | `POST /mcp`, `GET /health` | `Authorization: Bearer` (fail closed) |
+
+Deploy separately so browser traffic and agent traffic can bind different AI Search instances if you want.
+
+```sh
+npm run deploy       # query Worker
+npm run deploy:mcp   # MCP Worker
+wrangler secret put MCP_TOKEN -c wrangler.mcp.toml
+wrangler secret put TURNSTILE_SECRET   # optional; skips verification when unset
+```
+
+### MCP client wiring
+
+```json
+{
+  "mcpServers": {
+    "search-mcp": {
+      "type": "http",
+      "url": "https://YOUR_MCP_HOST/mcp",
+      "headers": { "Authorization": "Bearer YOUR_TOKEN" }
+    }
+  }
+}
+```
+
+`MCP_TOKEN` accepts a single token or comma-separated `name=token` pairs for per-consumer attribution in logs.
+
+## Corpus sync
+
+```sh
+export R2_ACCESS_KEY_ID=... R2_SECRET_ACCESS_KEY=... CLOUDFLARE_ACCOUNT_ID=...
+export CORPUS_GIT_ORG=your-org GITHUB_TOKEN=...   # for sync-runner clone auth
+
+npm run sync:dry     # plan upload for the default `corpus` target
+npm run sync         # upload + prune
+npm run sync:run     # isolated clone root, sync all targets, optional reindex
+```
+
+The sync remaps non-native extensions (`.ts`, `.tsx`, extensionless `Dockerfile`, `.service`, etc.) to `.txt` keys so AI Search indexes them. See `scripts/sync-ingest.mjs`.
+
+If you name a target `public`, optional boundary checks refuse overlap with `restrictedRepos` and can verify GitHub repo visibility before upload.
+
+## Ask widget
+
+Copy `public/ask-widget.js` and `public/ask-widget.css` to your docs site:
+
+```html
+<div id="docs-ask"></div>
+<script defer src="/ask-widget.js"
+        data-endpoint="https://search.example.com/ask"
+        data-target="#docs-ask"
+        data-label="Ask the docs"
+        data-sitekey="YOUR_TURNSTILE_SITEKEY"></script>
+```
+
+## Who this is for
+
+Operators building documentation search, agent tooling, or internal knowledge bases on [Cloudflare AI Search](https://developers.cloudflare.com/ai-search/) with MCP and a browser widget.
+
+## Links
+
+- **Deploy guide:** [docs/DEPLOY.md](docs/DEPLOY.md)
+- **Skyphusion Labs:** https://skyphusion.org · **Org:** https://github.com/skyphusion-labs
+- **Related:** [vivijure](https://github.com/skyphusion-labs/vivijure), [postern](https://github.com/skyphusion-labs/postern)
+
+## License
+
+AGPL-3.0-only. See [LICENSE](LICENSE).
+
+## Community
+
+- [Contributing](CONTRIBUTING.md)
+- [Security policy](SECURITY.md)
+- [Code of Conduct](CODE_OF_CONDUCT.md)

package/package.json

@@ -0,0 +1,75 @@
+{
+  "name": "@skyphusion/search-mcp",
+  "version": "0.1.2",
+  "description": "Bearer-gated MCP server and streaming query Worker over Cloudflare AI Search, plus a git-to-R2 corpus sync toolkit.",
+  "license": "AGPL-3.0-only",
+  "author": "Conrad Rockenhaus <conrad@skyphusion.org>",
+  "homepage": "https://github.com/skyphusion-labs/search-mcp#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/skyphusion-labs/search-mcp.git"
+  },
+  "bugs": {
+    "url": "https://github.com/skyphusion-labs/search-mcp/issues"
+  },
+  "keywords": [
+    "cloudflare-workers",
+    "mcp",
+    "ai-search",
+    "rag",
+    "semantic-search",
+    "model-context-protocol",
+    "self-hosted",
+    "cloudflare",
+    "corpus-sync"
+  ],
+  "type": "module",
+  "engines": {
+    "node": ">=20"
+  },
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/"
+  },
+  "files": [
+    "scripts/corpus-boundary.mjs",
+    "scripts/config-paths.mjs",
+    "scripts/sync-ingest.mjs",
+    "scripts/sync.mjs",
+    "scripts/sync-runner.mjs",
+    "scripts/targets.json.example",
+    "public/ask-widget.js",
+    "public/ask-widget.css",
+    "LICENSE",
+    "README.md"
+  ],
+  "bin": {
+    "search-mcp-sync": "./scripts/sync.mjs",
+    "search-mcp-sync-run": "./scripts/sync-runner.mjs"
+  },
+  "scripts": {
+    "dev": "wrangler dev",
+    "dev:mcp": "wrangler dev -c wrangler.mcp.toml",
+    "deploy": "wrangler deploy",
+    "deploy:mcp": "wrangler deploy -c wrangler.mcp.toml",
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run",
+    "prepack": "npm run typecheck && npm test",
+    "sync": "node scripts/sync.mjs corpus",
+    "sync:public": "node scripts/sync.mjs public",
+    "sync:internal": "node scripts/sync.mjs internal",
+    "sync:dry": "node scripts/sync.mjs corpus --dry-run",
+    "sync:run": "node scripts/sync-runner.mjs",
+    "materialize-config": "node scripts/materialize-config.mjs"
+  },
+  "dependencies": {
+    "@aws-sdk/client-s3": "^3.888.0"
+  },
+  "devDependencies": {
+    "@cloudflare/workers-types": "^4.20260606.1",
+    "@vitest/coverage-istanbul": "^4.1.10",
+    "typescript": "^6.0.3",
+    "vitest": "^4.1.10",
+    "wrangler": "^4.103.0"
+  }
+}

package/public/ask-widget.css

@@ -0,0 +1,77 @@
+/* Embeddable ask widget. Neutral defaults; override any .vjask-* rule on the host site. */
+.vjask {
+  --vjask-fg: #e8ecf4;
+  --vjask-muted: #93a0b5;
+  --vjask-bg: rgba(255, 255, 255, 0.03);
+  --vjask-border: rgba(255, 255, 255, 0.14);
+  --vjask-accent: #22d3ee;
+  --vjask-radius: 14px;
+  font-family: inherit;
+  color: var(--vjask-fg);
+}
+.vjask-form {
+  display: flex;
+  flex-direction: column;
+  gap: 0.75rem;
+}
+.vjask-label {
+  font-weight: 700;
+  font-size: 0.95rem;
+}
+.vjask-row {
+  display: flex;
+  gap: 0.5rem;
+}
+.vjask-input {
+  flex: 1;
+  min-width: 0;
+  padding: 0.7rem 0.9rem;
+  border-radius: var(--vjask-radius);
+  border: 1px solid var(--vjask-border);
+  background: var(--vjask-bg);
+  color: var(--vjask-fg);
+  font: inherit;
+}
+.vjask-input:focus {
+  outline: none;
+  border-color: var(--vjask-accent);
+}
+.vjask-btn {
+  padding: 0.7rem 1.2rem;
+  border-radius: var(--vjask-radius);
+  border: 1px solid transparent;
+  background: var(--vjask-accent);
+  color: #05121a;
+  font: inherit;
+  font-weight: 700;
+  cursor: pointer;
+}
+.vjask-btn:disabled {
+  opacity: 0.55;
+  cursor: progress;
+}
+.vjask-answer {
+  white-space: pre-wrap;
+  line-height: 1.7;
+  min-height: 1.5rem;
+  color: var(--vjask-fg);
+}
+.vjask-sources {
+  list-style: none;
+  margin: 0;
+  padding: 0.75rem 0 0;
+  border-top: 1px solid var(--vjask-border);
+  font-size: 0.85rem;
+  color: var(--vjask-muted);
+  display: flex;
+  flex-wrap: wrap;
+  gap: 0.35rem 0.9rem;
+}
+.vjask-sources-title {
+  width: 100%;
+  font-weight: 700;
+  color: var(--vjask-fg);
+}
+.vjask-sources li {
+  font-family: ui-monospace, "JetBrains Mono", monospace;
+}

package/public/ask-widget.js

@@ -0,0 +1,165 @@
+// Embeddable ask widget. Vanilla JS, no build step. Streams answers from POST /ask.
+//
+// Usage:
+//   <div id="docs-ask"></div>
+//   <script defer src="/ask-widget.js"
+//           data-endpoint="https://search.example.com/ask"
+//           data-target="#docs-ask"
+//           data-label="Ask the docs"
+//           data-placeholder="How do I deploy?"
+//           data-sitekey="<TURNSTILE_SITEKEY>"></script>
+(function () {
+  "use strict";
+  var script = document.currentScript;
+  var endpoint = script.getAttribute("data-endpoint") || "/ask";
+  var targetSel = script.getAttribute("data-target") || "#docs-ask";
+  var label = script.getAttribute("data-label") || "Ask";
+  var placeholder = script.getAttribute("data-placeholder") || "Ask a question about the docs";
+  var sitekey = script.getAttribute("data-sitekey") || "";
+
+  function ready(fn) {
+    if (document.readyState !== "loading") fn();
+    else document.addEventListener("DOMContentLoaded", fn);
+  }
+
+  ready(function () {
+    var root = document.querySelector(targetSel);
+    if (!root) return;
+    root.classList.add("vjask");
+    // Static template only. Label, placeholder, and Turnstile sitekey are applied via
+    // the DOM API below so attribute values are never concatenated into innerHTML.
+    root.innerHTML =
+      '<form class="vjask-form">' +
+      '  <label class="vjask-label" for="vjask-input"></label>' +
+      '  <div class="vjask-row">' +
+      '    <input id="vjask-input" class="vjask-input" type="text" autocomplete="off" maxlength="2000" />' +
+      '    <button class="vjask-btn" type="submit">Ask</button>' +
+      "  </div>" +
+      (sitekey ? '  <div class="vjask-turnstile-slot"></div>' : "") +
+      '  <div class="vjask-answer" aria-live="polite"></div>' +
+      '  <ul class="vjask-sources" hidden></ul>' +
+      "</form>";
+
+    var form = root.querySelector(".vjask-form");
+    var input = root.querySelector(".vjask-input");
+    var btn = root.querySelector(".vjask-btn");
+    var answer = root.querySelector(".vjask-answer");
+    var sources = root.querySelector(".vjask-sources");
+    var labelEl = root.querySelector(".vjask-label");
+    if (labelEl) labelEl.textContent = label;
+    if (input) input.setAttribute("placeholder", placeholder);
+
+    if (sitekey) {
+      var slot = root.querySelector(".vjask-turnstile-slot");
+      if (slot) {
+        var ts = document.createElement("div");
+        ts.className = "vjask-turnstile cf-turnstile";
+        ts.setAttribute("data-sitekey", sitekey);
+        ts.setAttribute("data-size", "flexible");
+        slot.replaceWith(ts);
+      }
+    }
+
+    function turnstileToken() {
+      if (!sitekey || !window.turnstile) return "";
+      try {
+        return window.turnstile.getResponse() || "";
+      } catch (e) {
+        return "";
+      }
+    }
+
+    function renderSources(chunks) {
+      var seen = {};
+      var items = [];
+      (chunks || []).forEach(function (c) {
+        var key = c && c.item && c.item.key;
+        if (key && !seen[key]) {
+          seen[key] = true;
+          items.push(key);
+        }
+      });
+      if (!items.length) return;
+      sources.innerHTML = "<li class='vjask-sources-title'>Sources</li>" +
+        items.map(function (k) {
+          return "<li>" + k.replace(/[&<>]/g, function (ch) {
+            return { "&": "&amp;", "<": "&lt;", ">": "&gt;" }[ch];
+          }) + "</li>";
+        }).join("");
+      sources.hidden = false;
+    }
+
+    function handleEvent(block) {
+      var isChunks = /(^|\n)event:\s*chunks/.test(block);
+      var m = block.match(/(^|\n)data:\s?(.*)$/s);
+      if (!m) return;
+      var payload = m[2].trim();
+      if (payload === "[DONE]") return;
+      var data;
+      try {
+        data = JSON.parse(payload);
+      } catch (e) {
+        return;
+      }
+      if (isChunks) {
+        renderSources(data);
+        return;
+      }
+      var delta = data && data.choices && data.choices[0] && data.choices[0].delta;
+      if (delta && delta.content) answer.textContent += delta.content;
+    }
+
+    form.addEventListener("submit", function (ev) {
+      ev.preventDefault();
+      var q = input.value.trim();
+      if (!q) return;
+      btn.disabled = true;
+      answer.textContent = "";
+      sources.hidden = true;
+      sources.innerHTML = "";
+
+      fetch(endpoint, {
+        method: "POST",
+        headers: { "content-type": "application/json" },
+        body: JSON.stringify({ question: q, turnstileToken: turnstileToken() }),
+      })
+        .then(function (res) {
+          if (!res.ok || !res.body) {
+            return res.json().then(
+              function (j) {
+                throw new Error(j.error || "error " + res.status);
+              },
+              function () {
+                throw new Error("error " + res.status);
+              },
+            );
+          }
+          var reader = res.body.getReader();
+          var decoder = new TextDecoder();
+          var buf = "";
+          function pump() {
+            return reader.read().then(function (r) {
+              if (r.done) return;
+              buf += decoder.decode(r.value, { stream: true });
+              var parts = buf.split("\n\n");
+              buf = parts.pop();
+              parts.forEach(handleEvent);
+              return pump();
+            });
+          }
+          return pump();
+        })
+        .catch(function (err) {
+          answer.textContent = "Sorry, something went wrong (" + err.message + ").";
+        })
+        .finally(function () {
+          btn.disabled = false;
+          if (sitekey && window.turnstile) {
+            try {
+              window.turnstile.reset();
+            } catch (e) {}
+          }
+        });
+    });
+  });
+})();

package/scripts/config-paths.mjs

@@ -0,0 +1,40 @@
+import { existsSync } from "node:fs";
+import { basename, dirname, join } from "node:path";
+
+/** Resolve targets.json for repo dev or npm-installed CLI usage. */
+export function resolveTargetsPath(scriptsDir) {
+  if (process.env.SEARCH_MCP_TARGETS) {
+    return process.env.SEARCH_MCP_TARGETS;
+  }
+  const candidates = [
+    join(process.cwd(), "targets.json"),
+    join(scriptsDir, "targets.json"),
+  ];
+  for (const path of candidates) {
+    if (existsSync(path)) return path;
+  }
+  return null;
+}
+
+/** Default git clone root for corpus sync. */
+export function defaultRepoRoot(scriptsDir, targetsPath) {
+  if (process.env.SYNC_REPO_ROOT) {
+    return process.env.SYNC_REPO_ROOT;
+  }
+  if (!targetsPath) {
+    return join(scriptsDir, "..", "..");
+  }
+  const dir = dirname(targetsPath);
+  if (dir === scriptsDir) {
+    return join(scriptsDir, "..", "..");
+  }
+  return dir;
+}
+
+export function targetsHelp(scriptsDir) {
+  const example = join(scriptsDir, "targets.json.example");
+  return [
+    "Missing targets.json.",
+    `Copy ${example} to ./targets.json (or set SEARCH_MCP_TARGETS).`,
+  ].join(" ");
+}

package/scripts/corpus-boundary.mjs

@@ -0,0 +1,131 @@
+// Optional safety checks for scripts/sync.mjs when a target is named `public`.
+//
+// (1) targets.json overlap: assert targets.public.repos ∩ restrictedRepos === ∅.
+//     restrictedRepos only drives optional per-file metadata; it does NOT exclude a
+//     repo from the public bucket.
+// (2) optional live GitHub visibility: refuse any non-public repo before upload.
+
+/** Pure: repos listed in BOTH the public target and restrictedRepos. */
+export function publicRestrictedOverlap(publicRepos, restrictedRepos) {
+  const restricted = new Set(restrictedRepos || []);
+  return (publicRepos || []).filter((r) => restricted.has(r));
+}
+
+/** Fail-closed when target is public and restrictedRepos overlaps the public list. */
+export function assertPublicCorpusBoundary(cfg, targetName) {
+  if (targetName !== "public") return;
+  const overlap = publicRestrictedOverlap(
+    cfg?.targets?.public?.repos,
+    cfg?.restrictedRepos,
+  );
+  if (!overlap.length) return;
+  throw new Error(
+    "public corpus boundary violation: " +
+      overlap.join(", ") +
+      " listed in BOTH targets.public.repos and restrictedRepos. " +
+      "restrictedRepos only sets metadata; it does NOT exclude a repo from the public bucket. " +
+      "Remove each repo from one list before syncing public.",
+  );
+}
+
+function githubTokenFromEnv(env = process.env) {
+  return env.GITHUB_TOKEN || env.GH_TOKEN || "";
+}
+
+// GitHub repo/org slug charset; rejects path segments and other URL metacharacters
+// before repo names from targets.json reach the GitHub API fetch.
+const GITHUB_SLUG_RE = /^[A-Za-z0-9._-]{1,100}$/;
+
+export function assertGithubSlug(slug, kind) {
+  if (typeof slug !== "string" || !GITHUB_SLUG_RE.test(slug)) {
+    throw new Error(`invalid GitHub ${kind} for API request: ${String(slug)}`);
+  }
+  return slug;
+}
+
+const GITHUB_API_HEADERS = (token) => ({
+  Accept: "application/vnd.github+json",
+  Authorization: `Bearer ${token}`,
+  "X-GitHub-Api-Version": "2022-11-28",
+  "User-Agent": "search-mcp-corpus-boundary",
+});
+
+/** Paginated org repo list; URL uses only the org slug (env), never targets.json repo names. */
+async function fetchPublicRepoNames(org, token, fetchImpl) {
+  const names = new Set();
+  let page = 1;
+  while (true) {
+    const url = new URL(`/orgs/${org}/repos`, "https://api.github.com");
+    url.searchParams.set("type", "public");
+    url.searchParams.set("per_page", "100");
+    url.searchParams.set("page", String(page));
+    const res = await fetchImpl(url.toString(), { headers: GITHUB_API_HEADERS(token) });
+    if (res.status === 404) {
+      throw new Error(`GitHub org not found: ${org}`);
+    }
+    if (!res.ok) {
+      const snippet = (await res.text()).slice(0, 200);
+      throw new Error(`GitHub API ${res.status} listing public repos for ${org}: ${snippet}`);
+    }
+    const body = await res.json();
+    if (!Array.isArray(body) || !body.length) break;
+    for (const entry of body) {
+      if (entry && typeof entry.name === "string") names.add(entry.name);
+    }
+    if (body.length < 100) break;
+    page += 1;
+  }
+  return names;
+}
+
+/** Live check: each repo must be visibility=public on GitHub. */
+export async function verifyGithubRepoVisibility(repos, opts = {}) {
+  const {
+    org = "",
+    token = "",
+    fetchImpl = globalThis.fetch,
+  } = opts;
+  if (!org) {
+    throw new Error("org is required for GitHub visibility verification");
+  }
+  assertGithubSlug(org, "org");
+  if (!token) {
+    return { skipped: true, reason: "no_github_token", checked: [], nonPublic: [] };
+  }
+  if (!fetchImpl) {
+    throw new Error("fetch is not available for GitHub visibility verification");
+  }
+
+  const nonPublic = [];
+  const checked = [];
+  const publicNames = await fetchPublicRepoNames(org, token, fetchImpl);
+  for (const repo of repos || []) {
+    assertGithubSlug(repo, "repo");
+    checked.push(repo);
+    if (!publicNames.has(repo)) {
+      nonPublic.push({
+        repo,
+        visibility: "not_public_or_missing",
+      });
+    }
+  }
+  return { skipped: false, checked, nonPublic };
+}
+
+/** Fail-closed when any public-target repo is not public on GitHub. Skips when no token. */
+export async function assertPublicGithubVisibility(cfg, opts = {}) {
+  const repos = cfg?.targets?.public?.repos || [];
+  const org = opts.org ?? process.env.CORPUS_GIT_ORG ?? "";
+  const token = opts.token ?? githubTokenFromEnv(opts.env);
+  const result = await verifyGithubRepoVisibility(repos, { ...opts, org, token });
+  if (result.skipped) return result;
+  if (!result.nonPublic.length) return result;
+  const detail = result.nonPublic
+    .map((x) => `${x.repo} (${x.visibility}${x.reason ? ": " + x.reason : ""})`)
+    .join(", ");
+  throw new Error(
+    "public corpus boundary violation: non-public repos on GitHub would sync to the public bucket: " +
+      detail +
+      ". Flip public on GitHub, remove from targets.public.repos, or sync under a non-public target.",
+  );
+}