@spicrawl/mcp 0.1.1 → 0.1.3

package/CHANGELOG.md

@@ -3,6 +3,18 @@
 All notable changes to `@spicrawl/mcp` are documented here. This project follows
 [Semantic Versioning](https://semver.org/).
 
+## 0.1.3 — 2026-10-01
+
+- `spicrawl_scrape` on the ChatGPT endpoint (`/chatgpt/mcp`) is now annotated `readOnlyHint: true`. That surface only ever sends an HTTP GET with no method, body, headers or actions, so it cannot change anything on a site. On `/mcp` it stays not read-only, because `method` and `actions` can submit forms.
+- `spicrawl_batch_results` on the ChatGPT endpoint now returns page content even when it is called before the job is `completed`. The API inlines `result.content` only once the whole job is terminal, while each item reports `succeeded` as soon as it finishes, so a model that read results early got metadata only. A succeeded item without content is now filled from `GET /v1/batch/{id}/tasks/{seq}/content` (at most 25 items and 2 MiB per page). An item that can't be read yet is marked `unavailable`, and one past the size cap is marked `omitted`.
+
+## 0.1.2 — 2026-10-01
+
+- Every tool's MCP annotations are now accurate, and a tool without a `title` and explicit `readOnlyHint`, `destructiveHint` and `openWorldHint` no longer compiles. `spicrawl_scrape` is no longer read-only, because `method` and `actions` can submit forms. `spicrawl_session_release` is destructive, because it purges cookies. Tools confined to your own account are no longer open-world. `test/annotations.test.mjs` pins each value.
+- `SPICRAWL_MCP_HIDE_UNAVAILABLE` (off by default) leaves out what is announced but not available yet: `spicrawl_browser_connect_url` is not registered and every "Coming soon" argument is dropped from the schemas.
+- Tool descriptions no longer name another product. The scrape description states billing and side effects. A printed PDF's JSON no longer carries the request id.
+- A ChatGPT endpoint on the HTTP server, off unless `SPICRAWL_CHATGPT_ENABLED` is on: stateless `POST /chatgpt/mcp` that takes OAuth access tokens only (introspected at `<SPICRAWL_OAUTH_ISSUER>/api/oauth/introspect`, positive answers cached at most 30 s), protected-resource metadata at `/.well-known/oauth-protected-resource/chatgpt/mcp` (and at the root with `SPICRAWL_CHATGPT_ROOT_PRM`), the OpenAI domain challenge at `/.well-known/openai-apps-challenge`, and a restricted six-tool profile with OAuth `securitySchemes` and sanitized results. `/mcp` is unchanged.
+
 ## 0.1.1 — 2026-10-01
 
 Documentation and package metadata only; the server and its 25 tools are unchanged.

package/README.md

@@ -61,7 +61,7 @@ Both transports serve the same 25 `spicrawl_*` tools, and every call runs under
 
 ## Tools
 
-25 tools, grouped by task. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Full argument reference: [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp#tools).
+25 tools, grouped by task. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a `title` and explicit `readOnlyHint`, `destructiveHint` and `openWorldHint` annotations, so a client knows which calls to confirm: `spicrawl_scrape` is not read-only (its `method` and `actions` can submit forms), and `spicrawl_batch_cancel`, `spicrawl_session_release` and `spicrawl_session_delete` are destructive. Full argument reference: [docs.spicrawl.com/agents/mcp](https://docs.spicrawl.com/agents/mcp#tools).
 
 ### Scrape a website to Markdown
 
@@ -114,7 +114,7 @@ Both transports serve the same 25 `spicrawl_*` tools, and every call runs under
 
 ### Coming soon
 
-`spicrawl_browser_connect_url` will mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from Puppeteer or Playwright; remote browsers are not available yet. The schemas also accept `ai_extract`, `stealth`, `extract_preset`, `premium_proxy`, `proxy_country` and `sticky_key` ahead of launch. Agents should not send them yet.
+`spicrawl_browser_connect_url` will mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from Puppeteer or Playwright; remote browsers are not available yet. The schemas also accept `ai_extract`, `stealth`, `extract_preset`, `premium_proxy`, `proxy_country` and `sticky_key` ahead of launch. Agents should not send them yet. To list only what works today, set `SPICRAWL_MCP_HIDE_UNAVAILABLE=1` (see [Environment variables](https://github.com/Spicrawl/mcp#environment-variables)).
 
 ## Example prompts
 
@@ -335,6 +335,14 @@ A stdio-only client runs `npx -y @spicrawl/mcp` with `SPICRAWL_API_KEY` in its e
 | `SPICRAWL_DOCS_HOST` | none | both | Legacy: an origin whose docs are at `/docs`. Read only when `SPICRAWL_DOCS_URL` is unset. |
 | `SPICRAWL_MCP_ADDR` | `127.0.0.1:8090` | HTTP | Listen address, `host:port`. |
 | `SPICRAWL_MCP_SESSION_IDLE_MS` | `1800000` (30 min) | HTTP | Idle MCP sessions are dropped after this long. |
+| `SPICRAWL_MCP_HIDE_UNAVAILABLE` | off | both | `1`, `true`, `yes` or `on` leaves out what is announced but not available yet: `spicrawl_browser_connect_url` is not registered (24 tools) and every argument marked "Coming soon" is dropped from the schemas (`ai_extract`, `stealth`, `extract_preset`, `premium_proxy`, `proxy_country`, `sticky_key`, and the session `rotate_ip`, `region_pool` and `fingerprint`). The `camoufox` engine value stays in the `engine` enums. Read when the server starts a session, so restart the process after changing it. |
+| `SPICRAWL_CHATGPT_ENABLED` | off | HTTP | `1`, `true`, `yes` or `on` turns on the [ChatGPT endpoint](#chatgpt-endpoint-oauth). Off, `/chatgpt/mcp`, its metadata and the OpenAI challenge all answer `404`. |
+| `SPICRAWL_CHATGPT_RESOURCE` | `https://mcp.spicrawl.com/chatgpt/mcp` | HTTP | The OAuth resource identifier. Tokens must be issued for exactly this `aud`; the metadata is published at `/.well-known/oauth-protected-resource` + its path. |
+| `SPICRAWL_OAUTH_ISSUER` | `https://app.spicrawl.com` | HTTP | The authorization server, listed in the metadata. Tokens are introspected at `<issuer>/api/oauth/introspect`. |
+| `SPICRAWL_OAUTH_INTROSPECT_SECRET` | none | HTTP | This server's bearer secret at the introspection endpoint. Unset, every token is answered `503`. |
+| `SPICRAWL_CHATGPT_ROOT_PRM` | off | HTTP | Also serve the metadata at the root `/.well-known/oauth-protected-resource`. Off by default because `/mcp` takes API keys and must not advertise OAuth. |
+| `SPICRAWL_OPENAI_APPS_CHALLENGE` | none | HTTP | The OpenAI domain-verification token served at `/.well-known/openai-apps-challenge`. |
+| `SPICRAWL_OPENAI_APPS_CHALLENGE_FILE` | none | HTTP | A file holding that token (surrounding whitespace and the trailing newline are dropped). Read when `SPICRAWL_OPENAI_APPS_CHALLENGE` is unset. |
 
 With no docs variable set, a server pointed at a self-hosted API reads that API's own docs at `<SPICRAWL_PUBLIC_BASE_URL or SPICRAWL_BASE_URL>/docs`.
 
@@ -353,6 +361,42 @@ npx -y @spicrawl/mcp --http
 - Each caller sends their own key as `Authorization: Bearer spicrawl_live_...`. The key is checked against the API before a session opens (`401` if refused, `503` with `Retry-After` if the API cannot be reached), and a session is pinned to the key that opened it.
 - The server speaks plain HTTP. Put it behind TLS before sending keys across a network.
 
+### ChatGPT endpoint (OAuth)
+
+The same process can serve a second, restricted MCP endpoint for ChatGPT apps. It is off unless `SPICRAWL_CHATGPT_ENABLED` is on, and `/mcp` is unchanged either way.
+
+| Route | What it does |
+|---|---|
+| `POST /chatgpt/mcp` | MCP Streamable HTTP, **stateless**: no `Mcp-Session-Id`, a fresh server per request, JSON responses. Other methods answer `405`. |
+| `GET /.well-known/oauth-protected-resource/chatgpt/mcp` | Protected-resource metadata (RFC 9728): `resource`, `authorization_servers` (the issuer), `scopes_supported: ["scrape","batch"]`, `resource_documentation`. |
+| `GET /.well-known/oauth-protected-resource` | The same document, only with `SPICRAWL_CHATGPT_ROOT_PRM`. |
+| `GET /.well-known/openai-apps-challenge` | The verification token as `text/plain`, exact bytes, no trailing newline; `404` when none is configured. |
+
+Authentication takes OAuth access tokens (`spicrawl_oat_…`) only:
+
+- No token: `401` with `WWW-Authenticate: Bearer resource_metadata="https://mcp.spicrawl.com/.well-known/oauth-protected-resource/chatgpt/mcp", scope="scrape batch"`.
+- A bad, inactive, expired or wrong-audience token, or an API key (`spicrawl_live_…`/`spicrawl_test_…`, never accepted here): the same `401` plus `error="invalid_token", error_description="…"`.
+- A tool call the token's scopes do not cover: `403` with `error="insufficient_scope"`. `spicrawl_scrape` needs `scrape`, the batch tools need `batch`, the docs tools need only a valid token.
+- The authorization server unreachable or refusing this server's secret: `503` with `Retry-After`.
+
+Each token is checked with `POST <issuer>/api/oauth/introspect` (`Authorization: Bearer <SPICRAWL_OAUTH_INTROSPECT_SECRET>`, form body `token=…`). Only `active: true` with `aud` equal to the resource is accepted. A positive answer is cached by the token's SHA-256 for at most 30 seconds and never past its `exp`. Tools run upstream under the `api_key` the introspection returns; neither it nor the token is logged. If the API refuses that key mid-call, the tool returns an error whose `_meta["mcp/www_authenticate"]` carries a fresh `invalid_token` challenge, and the token is dropped from the cache.
+
+The endpoint lists six tools, each with explicit `readOnlyHint`/`destructiveHint`/`openWorldHint`, a title, and `securitySchemes: [{type: "oauth2", scopes: [...]}]` (also in `_meta.securitySchemes`):
+
+| Tool | Scopes | Arguments |
+|---|---|---|
+| `spicrawl_scrape` | `scrape` | `url` (http/https only), `format` (`markdown`/`text`/`html`/`json`), `render`, `main_content_only`, `include_tags`, `exclude_tags`, `links`, `extract` (CSS selectors), `autoparse`, `wait_for`, `cache`, `cache_ttl`, `max_cost` (default 5). Always an HTTP GET. |
+| `spicrawl_batch_submit` | `batch` | `urls` (1 to 25), `format`, `render`, `main_content_only`, `max_cost`, `credit_budget` (default 50). |
+| `spicrawl_batch_status` | `batch` | `job_id`. |
+| `spicrawl_batch_results` | `batch` | `job_id`, `status`, `limit` (default 25, max 100), `cursor`. |
+| `spicrawl_docs_search`, `spicrawl_docs_read` | none | As on `/mcp`. |
+
+`spicrawl_scrape` is read-only here, unlike on `/mcp`: this endpoint only sends a GET, with no `method`, body, headers or `actions`. `spicrawl_batch_submit` is not read-only, because it creates a job.
+
+`spicrawl_batch_results` fills in the content of a succeeded item that the results page lists without it, which happens while the job is still finishing, from `GET /v1/batch/{id}/tasks/{seq}/content`. It fetches at most 25 items and 2 MiB per page. An item that can't be read yet comes back `unavailable`, and one past the size cap `omitted`.
+
+Every other argument is refused, not dropped. Results keep the content, the site's status and the credits charged; request ids, timestamps, engine and proxy details, storage references and project ids are left out.
+
 Installed globally (`npm install -g @spicrawl/mcp`), the command is `spicrawl-mcp` (alias `spicrawl-mcp-server`), with `--http`, `--version` and `--help`.
 
 ## FAQ
@@ -373,7 +417,7 @@ Yes, and with Claude Desktop, Windsurf, Gemini CLI, Codex and any client that sp
 There is no single crawl tool. Scrape a page with `links: true` to get its links, then feed them to an open batch job (`open: true` and `spicrawl_batch_add_items`).
 
 ### Does it support OAuth?
-No. The server accepts Spicrawl API keys only; turn OAuth off for this server in clients that try it on a `401`.
+Not on `/mcp`: it accepts Spicrawl API keys only, so turn OAuth off for it in clients that try it on a `401`. The separate [ChatGPT endpoint](#chatgpt-endpoint-oauth) at `/chatgpt/mcp` takes OAuth access tokens only, when the operator turns it on.
 
 ### Should I use the MCP server, the SDK or the CLI?
 Use the MCP server when an AI agent should call Spicrawl as tools. Use [`@spicrawl/sdk`](https://github.com/Spicrawl/sdk) in your own TypeScript or JavaScript code, [`@spicrawl/cli`](https://github.com/Spicrawl/cli) in a terminal or shell script, and the [REST API](https://docs.spicrawl.com/quickstart) from any other language.

package/dist/chatgpt/http.d.ts

@@ -0,0 +1,22 @@
+/**
+ * The ChatGPT surface's HTTP routes, mounted by http.ts when
+ * SPICRAWL_CHATGPT_ENABLED is on (all of them answer 404 otherwise):
+ *
+ *   POST /chatgpt/mcp                                     stateless MCP, OAuth access tokens only
+ *   GET  /.well-known/oauth-protected-resource/chatgpt/mcp  protected-resource metadata (RFC 9728)
+ *   GET  /.well-known/oauth-protected-resource            the same, only with SPICRAWL_CHATGPT_ROOT_PRM
+ *   GET  /.well-known/openai-apps-challenge               OpenAI domain verification token
+ *
+ * Stateless: every POST is verified on its own and served by a fresh MCP server
+ * and transport, so no session can outlive or outrank the token that made it.
+ */
+import type { IncomingMessage, ServerResponse } from "node:http";
+import { type ChatgptConfig } from "./oauth.js";
+export declare const CHATGPT_MCP_PATH = "/chatgpt/mcp";
+export interface ChatgptDeps {
+    baseURL: string;
+    publicBaseURL: string;
+    log: (...a: unknown[]) => void;
+}
+/** Returns a router: true when it answered the request, false to let the caller 404 it. */
+export declare function chatgptRoutes(config: ChatgptConfig, deps: ChatgptDeps): (req: IncomingMessage, res: ServerResponse, path: string) => boolean;

package/dist/chatgpt/http.js

@@ -0,0 +1,127 @@
+/**
+ * The ChatGPT surface's HTTP routes, mounted by http.ts when
+ * SPICRAWL_CHATGPT_ENABLED is on (all of them answer 404 otherwise):
+ *
+ *   POST /chatgpt/mcp                                     stateless MCP, OAuth access tokens only
+ *   GET  /.well-known/oauth-protected-resource/chatgpt/mcp  protected-resource metadata (RFC 9728)
+ *   GET  /.well-known/oauth-protected-resource            the same, only with SPICRAWL_CHATGPT_ROOT_PRM
+ *   GET  /.well-known/openai-apps-challenge               OpenAI domain verification token
+ *
+ * Stateless: every POST is verified on its own and served by a fresh MCP server
+ * and transport, so no session can outlive or outrank the token that made it.
+ */
+import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
+import { ILayerClient } from "../client.js";
+import { KEY_RE, bearerToken, readJSON, rpcError } from "../httputil.js";
+import { ACCESS_TOKEN_RE, Introspector, SCOPES, bearerChallenge, prmPath, protectedResourceMetadata, } from "./oauth.js";
+import { TOOL_SCOPES, buildChatgptServer } from "./tools.js";
+export const CHATGPT_MCP_PATH = "/chatgpt/mcp";
+const ROOT_PRM_PATH = "/.well-known/oauth-protected-resource";
+const CHALLENGE_PATH = "/.well-known/openai-apps-challenge";
+/** Scopes a JSON-RPC message (or batch of them) needs: those of every tool it calls. */
+function requiredScopes(body) {
+    const out = new Set();
+    for (const msg of Array.isArray(body) ? body : [body]) {
+        if (!msg || typeof msg !== "object" || msg.method !== "tools/call")
+            continue;
+        const name = msg.params?.name;
+        for (const s of (typeof name === "string" && TOOL_SCOPES[name]) || [])
+            out.add(s);
+    }
+    return SCOPES.filter((s) => out.has(s));
+}
+/** Returns a router: true when it answered the request, false to let the caller 404 it. */
+export function chatgptRoutes(config, deps) {
+    const introspector = new Introspector(config, { log: deps.log });
+    const prm = prmPath(config.resource);
+    const metadata = JSON.stringify(protectedResourceMetadata(config));
+    const unauthorized = (res, description) => rpcError(res, 401, -32001, `Unauthorized: ${description ?? "connect with OAuth to use this endpoint"}`, {
+        "WWW-Authenticate": bearerChallenge(config.resource, description ? { error: "invalid_token", description } : {}),
+    });
+    async function handleMcp(req, res) {
+        const token = bearerToken(req);
+        if (!token)
+            return unauthorized(res);
+        if (KEY_RE.test(token)) {
+            return unauthorized(res, "API keys are not accepted on this endpoint; connect the Spicrawl app with OAuth");
+        }
+        if (!ACCESS_TOKEN_RE.test(token))
+            return unauthorized(res, "The access token is malformed");
+        const check = await introspector.introspect(token);
+        if (!check.ok) {
+            return check.status === 401
+                ? unauthorized(res, check.description)
+                : rpcError(res, 503, -32000, `Service Unavailable: ${check.description}; retry shortly`, { "Retry-After": "5" });
+        }
+        const grant = check.grant;
+        if (req.method !== "POST") {
+            return rpcError(res, 405, -32000, "Method not allowed: this endpoint is stateless; POST JSON-RPC requests", { Allow: "POST" });
+        }
+        let body;
+        try {
+            body = await readJSON(req);
+        }
+        catch {
+            return rpcError(res, 400, -32700, "Parse error: body must be JSON (max 4 MiB)");
+        }
+        const missing = requiredScopes(body).filter((s) => !grant.scopes.has(s));
+        if (missing.length) {
+            const description = `This tool needs the ${missing.join(" and ")} permission; reconnect the Spicrawl app to grant it`;
+            return rpcError(res, 403, -32003, `Forbidden: ${description}`, {
+                "WWW-Authenticate": bearerChallenge(config.resource, {
+                    scope: SCOPES.filter((s) => grant.scopes.has(s) || missing.includes(s)).join(" "),
+                    error: "insufficient_scope",
+                    description,
+                }),
+            });
+        }
+        const client = new ILayerClient({ apiKey: grant.apiKey, baseURL: deps.baseURL, publicBaseURL: deps.publicBaseURL });
+        const server = buildChatgptServer(client, {
+            upstreamChallenge: (scopes) => bearerChallenge(config.resource, {
+                scope: (scopes.length ? scopes : SCOPES).join(" "),
+                error: "invalid_token",
+                description: "The Spicrawl credential behind this access token was refused; reconnect the Spicrawl app",
+            }),
+            onUpstreamUnauthorized: () => introspector.forget(token),
+        });
+        const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined, enableJsonResponse: true });
+        res.on("close", () => {
+            void transport.close().catch(() => { });
+            void server.close().catch(() => { });
+        });
+        await server.connect(transport);
+        await transport.handleRequest(req, res, body);
+    }
+    const sendMetadata = (req, res) => {
+        res.writeHead(200, {
+            "Content-Type": "application/json",
+            "Cache-Control": "public, max-age=300",
+            "Access-Control-Allow-Origin": "*",
+        });
+        res.end(req.method === "HEAD" ? undefined : metadata);
+    };
+    return (req, res, path) => {
+        if (!config.enabled)
+            return false;
+        const read = req.method === "GET" || req.method === "HEAD";
+        if (path === CHATGPT_MCP_PATH) {
+            handleMcp(req, res).catch((err) => {
+                deps.log("chatgpt request failed:", err instanceof Error ? err.message : String(err));
+                rpcError(res, 500, -32603, "Internal error");
+            });
+            return true;
+        }
+        if (read && (path === prm || (config.rootPrm && path === ROOT_PRM_PATH))) {
+            sendMetadata(req, res);
+            return true;
+        }
+        if (read && path === CHALLENGE_PATH && config.appsChallenge) {
+            const bytes = Buffer.from(config.appsChallenge, "utf8");
+            res.writeHead(200, { "Content-Type": "text/plain", "Content-Length": String(bytes.length) });
+            res.end(req.method === "HEAD" ? undefined : bytes);
+            return true;
+        }
+        return false;
+    };
+}
+//# sourceMappingURL=http.js.map

package/dist/chatgpt/http.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"http.js","sourceRoot":"","sources":["../../src/chatgpt/http.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAGH,OAAO,EAAE,6BAA6B,EAAE,MAAM,oDAAoD,CAAC;AACnG,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,QAAQ,EAAE,QAAQ,EAAE,MAAM,gBAAgB,CAAC;AACzE,OAAO,EACL,eAAe,EACf,YAAY,EACZ,MAAM,EACN,eAAe,EACf,OAAO,EACP,yBAAyB,GAI1B,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,WAAW,EAAE,kBAAkB,EAAE,MAAM,YAAY,CAAC;AAE7D,MAAM,CAAC,MAAM,gBAAgB,GAAG,cAAc,CAAC;AAC/C,MAAM,aAAa,GAAG,uCAAuC,CAAC;AAC9D,MAAM,cAAc,GAAG,oCAAoC,CAAC;AAQ5D,wFAAwF;AACxF,SAAS,cAAc,CAAC,IAAa;IACnC,MAAM,GAAG,GAAG,IAAI,GAAG,EAAS,CAAC;IAC7B,KAAK,MAAM,GAAG,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC;QACtD,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ,IAAK,GAA4B,CAAC,MAAM,KAAK,YAAY;YAAE,SAAS;QACvG,MAAM,IAAI,GAAI,GAAuC,CAAC,MAAM,EAAE,IAAI,CAAC;QACnE,KAAK,MAAM,CAAC,IAAI,CAAC,OAAO,IAAI,KAAK,QAAQ,IAAI,WAAW,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE;YAAE,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;IACpF,CAAC;IACD,OAAO,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;AAC1C,CAAC;AAED,2FAA2F;AAC3F,MAAM,UAAU,aAAa,CAAC,MAAqB,EAAE,IAAiB;IACpE,MAAM,YAAY,GAAG,IAAI,YAAY,CAAC,MAAM,EAAE,EAAE,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;IACjE,MAAM,GAAG,GAAG,OAAO,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC;IACrC,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,yBAAyB,CAAC,MAAM,CAAC,CAAC,CAAC;IAEnE,MAAM,YAAY,GAAG,CAAC,GAAmB,EAAE,WAAoB,EAAE,EAAE,CACjE,QAAQ,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,iBAAiB,WAAW,IAAI,yCAAyC,EAAE,EAAE;QACtG,kBAAkB,EAAE,eAAe,CAAC,MAAM,CAAC,QAAQ,EAAE,WAAW,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,eAAe,EAAE,WAAW,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;KACjH,CAAC,CAAC;IAEL,KAAK,UAAU,SAAS,CAAC,GAAoB,EAAE,GAAmB;QAChE,MAAM,KAAK,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC;QAC/B,IAAI,CAAC,KAAK;YAAE,OAAO,YAAY,CAAC,GAAG,CAAC,CAAC;QACrC,IAAI,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;YACvB,OAAO,YAAY,CAAC,GAAG,EAAE,iFAAiF,CAAC,CAAC;QAC9G,CAAC;QACD,IAAI,CAAC,eAAe,CAAC,IAAI,CAAC,KAAK,CAAC;YAAE,OAAO,YAAY,CAAC,GAAG,EAAE,+BAA+B,CAAC,CAAC;QAE5F,MAAM,KAAK,GAAG,MAAM,YAAY,CAAC,UAAU,CAAC,KAAK,CAAC,CAAC;QACnD,IAAI,CAAC,KAAK,CAAC,EAAE,EAAE,CAAC;YACd,OAAO,KAAK,CAAC,MAAM,KAAK,GAAG;gBACzB,CAAC,CAAC,YAAY,CAAC,GAAG,EAAE,KAAK,CAAC,WAAW,CAAC;gBACtC,CAAC,CAAC,QAAQ,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,wBAAwB,KAAK,CAAC,WAAW,iBAAiB,EAAE,EAAE,aAAa,EAAE,GAAG,EAAE,CAAC,CAAC;QACrH,CAAC;QACD,MAAM,KAAK,GAAU,KAAK,CAAC,KAAK,CAAC;QAEjC,IAAI,GAAG,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;YAC1B,OAAO,QAAQ,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,wEAAwE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC,CAAC;QACjI,CAAC;QACD,IAAI,IAAa,CAAC;QAClB,IAAI,CAAC;YACH,IAAI,GAAG,MAAM,QAAQ,CAAC,GAAG,CAAC,CAAC;QAC7B,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,QAAQ,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,4CAA4C,CAAC,CAAC;QAClF,CAAC;QAED,MAAM,OAAO,GAAG,cAAc,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;QACzE,IAAI,OAAO,CAAC,MAAM,EAAE,CAAC;YACnB,MAAM,WAAW,GAAG,uBAAuB,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,qDAAqD,CAAC;YACtH,OAAO,QAAQ,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,cAAc,WAAW,EAAE,EAAE;gBAC7D,kBAAkB,EAAE,eAAe,CAAC,MAAM,CAAC,QAAQ,EAAE;oBACnD,KAAK,EAAE,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,IAAI,OAAO,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC;oBACjF,KAAK,EAAE,oBAAoB;oBAC3B,WAAW;iBACZ,CAAC;aACH,CAAC,CAAC;QACL,CAAC;QAED,MAAM,MAAM,GAAG,IAAI,YAAY,CAAC,EAAE,MAAM,EAAE,KAAK,CAAC,MAAM,EAAE,OAAO,EAAE,IAAI,CAAC,OAAO,EAAE,aAAa,EAAE,IAAI,CAAC,aAAa,EAAE,CAAC,CAAC;QACpH,MAAM,MAAM,GAAG,kBAAkB,CAAC,MAAM,EAAE;YACxC,iBAAiB,EAAE,CAAC,MAAM,EAAE,EAAE,CAC5B,eAAe,CAAC,MAAM,CAAC,QAAQ,EAAE;gBAC/B,KAAK,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC;gBAClD,KAAK,EAAE,eAAe;gBACtB,WAAW,EAAE,0FAA0F;aACxG,CAAC;YACJ,sBAAsB,EAAE,GAAG,EAAE,CAAC,YAAY,CAAC,MAAM,CAAC,KAAK,CAAC;SACzD,CAAC,CAAC;QACH,MAAM,SAAS,GAAG,IAAI,6BAA6B,CAAC,EAAE,kBAAkB,EAAE,SAAS,EAAE,kBAAkB,EAAE,IAAI,EAAE,CAAC,CAAC;QACjH,GAAG,CAAC,EAAE,CAAC,OAAO,EAAE,GAAG,EAAE;YACnB,KAAK,SAAS,CAAC,KAAK,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;YACvC,KAAK,MAAM,CAAC,KAAK,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;QACtC,CAAC,CAAC,CAAC;QACH,MAAM,MAAM,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;QAChC,MAAM,SAAS,CAAC,aAAa,CAAC,GAAG,EAAE,GAAG,EAAE,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,MAAM,YAAY,GAAG,CAAC,GAAoB,EAAE,GAAmB,EAAE,EAAE;QACjE,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE;YACjB,cAAc,EAAE,kBAAkB;YAClC,eAAe,EAAE,qBAAqB;YACtC,6BAA6B,EAAE,GAAG;SACnC,CAAC,CAAC;QACH,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC;IACxD,CAAC,CAAC;IAEF,OAAO,CAAC,GAAoB,EAAE,GAAmB,EAAE,IAAY,EAAW,EAAE;QAC1E,IAAI,CAAC,MAAM,CAAC,OAAO;YAAE,OAAO,KAAK,CAAC;QAClC,MAAM,IAAI,GAAG,GAAG,CAAC,MAAM,KAAK,KAAK,IAAI,GAAG,CAAC,MAAM,KAAK,MAAM,CAAC;QAE3D,IAAI,IAAI,KAAK,gBAAgB,EAAE,CAAC;YAC9B,SAAS,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE;gBAChC,IAAI,CAAC,GAAG,CAAC,yBAAyB,EAAE,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;gBACtF,QAAQ,CAAC,GAAG,EAAE,GAAG,EAAE,CAAC,KAAK,EAAE,gBAAgB,CAAC,CAAC;YAC/C,CAAC,CAAC,CAAC;YACH,OAAO,IAAI,CAAC;QACd,CAAC;QACD,IAAI,IAAI,IAAI,CAAC,IAAI,KAAK,GAAG,IAAI,CAAC,MAAM,CAAC,OAAO,IAAI,IAAI,KAAK,aAAa,CAAC,CAAC,EAAE,CAAC;YACzE,YAAY,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;YACvB,OAAO,IAAI,CAAC;QACd,CAAC;QACD,IAAI,IAAI,IAAI,IAAI,KAAK,cAAc,IAAI,MAAM,CAAC,aAAa,EAAE,CAAC;YAC5D,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,aAAa,EAAE,MAAM,CAAC,CAAC;YACxD,GAAG,CAAC,SAAS,CAAC,GAAG,EAAE,EAAE,cAAc,EAAE,YAAY,EAAE,gBAAgB,EAAE,MAAM,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;YAC7F,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;YACnD,OAAO,IAAI,CAAC;QACd,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC,CAAC;AACJ,CAAC"}

package/dist/chatgpt/oauth.d.ts

@@ -0,0 +1,100 @@
+/**
+ * OAuth for the ChatGPT surface (`/chatgpt/mcp`): its configuration, the
+ * protected-resource metadata it publishes (RFC 9728), the WWW-Authenticate
+ * challenges it answers with, and access-token introspection (RFC 7662) against
+ * the Spicrawl authorization server.
+ *
+ * The surface never takes an API key. A caller presents an OAuth access token
+ * (`spicrawl_oat_…`); the authorization server says which project it belongs to
+ * and hands back the API key to run that project's calls under. Neither the
+ * token nor that key is ever logged.
+ */
+/** Every scope the surface knows, in the order the challenge lists them. */
+export declare const SCOPES: readonly ["scrape", "batch"];
+export type Scope = (typeof SCOPES)[number];
+export declare const DEFAULT_RESOURCE = "https://mcp.spicrawl.com/chatgpt/mcp";
+export declare const DEFAULT_ISSUER = "https://app.spicrawl.com";
+export declare const RESOURCE_DOCUMENTATION = "https://docs.spicrawl.com/agents/mcp";
+/** OAuth access tokens the Spicrawl authorization server issues. */
+export declare const ACCESS_TOKEN_RE: RegExp;
+/** A positive introspection is reused for at most this long, and never past the token's `exp`. */
+export declare const CACHE_MAX_MS = 30000;
+export interface ChatgptConfig {
+    /** SPICRAWL_CHATGPT_ENABLED: the kill switch. Off: every route below answers 404. */
+    enabled: boolean;
+    /** SPICRAWL_CHATGPT_RESOURCE: the resource identifier tokens must be issued for (`aud`). */
+    resource: string;
+    /** SPICRAWL_OAUTH_ISSUER: the authorization server, no trailing slash. */
+    issuer: string;
+    /** SPICRAWL_OAUTH_INTROSPECT_SECRET: this server's credential at the introspection endpoint. */
+    introspectSecret: string;
+    /** SPICRAWL_CHATGPT_ROOT_PRM: also publish the metadata at the root well-known path. */
+    rootPrm: boolean;
+    /** SPICRAWL_OPENAI_APPS_CHALLENGE (or `…_FILE`): the domain-verification token, or "" for none. */
+    appsChallenge: string;
+}
+/** The configuration from the environment. Throws when a set value is unusable. */
+export declare function chatgptConfigFromEnv(env?: NodeJS.ProcessEnv): ChatgptConfig;
+/** Where the resource's metadata lives: `/.well-known/oauth-protected-resource` + the resource's path (RFC 9728 §3.1). */
+export declare function prmPath(resource: string): string;
+/** The metadata's absolute URL, as the challenge names it. */
+export declare function prmURL(resource: string): string;
+/** The protected-resource metadata document (RFC 9728). */
+export declare function protectedResourceMetadata(config: ChatgptConfig): {
+    resource: string;
+    authorization_servers: string[];
+    scopes_supported: ("scrape" | "batch")[];
+    resource_documentation: string;
+};
+/**
+ * A `Bearer` challenge (RFC 6750 §3, RFC 9728 §5.1). Without `error` it is the
+ * plain "authenticate here" answer to a request that sent no token.
+ */
+export declare function bearerChallenge(resource: string, opts?: {
+    scope?: string;
+    error?: "invalid_token" | "insufficient_scope";
+    description?: string;
+}): string;
+/** What an accepted token grants. */
+export interface Grant {
+    scopes: Set<string>;
+    /** The project's API key the token stands for; the upstream calls run under it. Never logged. */
+    apiKey: string;
+    clientId?: string;
+    /** Milliseconds since the epoch; undefined when the server sent no `exp`. */
+    expiresAt?: number;
+}
+export type Introspection = {
+    ok: true;
+    grant: Grant;
+} | {
+    ok: false;
+    status: 401;
+    description: string;
+} | {
+    ok: false;
+    status: 503;
+    description: string;
+};
+/**
+ * Introspects access tokens at `<issuer>/api/oauth/introspect`, caching
+ * positive answers by the token's SHA-256 for at most 30 s and never past the
+ * token's expiry. A negative answer is never cached, so a token issued a moment
+ * later is not refused from memory.
+ */
+export declare class Introspector {
+    private readonly config;
+    private readonly cache;
+    private readonly log;
+    constructor(config: Pick<ChatgptConfig, "issuer" | "resource" | "introspectSecret">, options?: {
+        log?: (...a: unknown[]) => void;
+        now?: () => number;
+    });
+    private now;
+    get endpoint(): string;
+    /** Forgets a token, e.g. once the API refused the key it stood for. */
+    forget(token: string): void;
+    introspect(token: string): Promise<Introspection>;
+    /** Drops expired entries, then the oldest ones, to stay under the cap. */
+    private evict;
+}

package/dist/chatgpt/oauth.js

@@ -0,0 +1,199 @@
+/**
+ * OAuth for the ChatGPT surface (`/chatgpt/mcp`): its configuration, the
+ * protected-resource metadata it publishes (RFC 9728), the WWW-Authenticate
+ * challenges it answers with, and access-token introspection (RFC 7662) against
+ * the Spicrawl authorization server.
+ *
+ * The surface never takes an API key. A caller presents an OAuth access token
+ * (`spicrawl_oat_…`); the authorization server says which project it belongs to
+ * and hands back the API key to run that project's calls under. Neither the
+ * token nor that key is ever logged.
+ */
+import { createHash } from "node:crypto";
+import { readFileSync } from "node:fs";
+import { USER_AGENT } from "../client.js";
+import { isTruthy } from "../server.js";
+import { KEY_RE } from "../httputil.js";
+/** Every scope the surface knows, in the order the challenge lists them. */
+export const SCOPES = ["scrape", "batch"];
+export const DEFAULT_RESOURCE = "https://mcp.spicrawl.com/chatgpt/mcp";
+export const DEFAULT_ISSUER = "https://app.spicrawl.com";
+export const RESOURCE_DOCUMENTATION = "https://docs.spicrawl.com/agents/mcp";
+/** OAuth access tokens the Spicrawl authorization server issues. */
+export const ACCESS_TOKEN_RE = /^spicrawl_oat_[A-Za-z0-9_-]{8,}$/;
+const INTROSPECT_TIMEOUT_MS = 5_000;
+/** A positive introspection is reused for at most this long, and never past the token's `exp`. */
+export const CACHE_MAX_MS = 30_000;
+const CACHE_MAX_ENTRIES = 10_000;
+const trimURL = (s) => (s ?? "").trim().replace(/\/+$/, "");
+/** The configuration from the environment. Throws when a set value is unusable. */
+export function chatgptConfigFromEnv(env = process.env) {
+    const resource = trimURL(env.SPICRAWL_CHATGPT_RESOURCE) || DEFAULT_RESOURCE;
+    const issuer = trimURL(env.SPICRAWL_OAUTH_ISSUER) || DEFAULT_ISSUER;
+    for (const [name, v] of [["SPICRAWL_CHATGPT_RESOURCE", resource], ["SPICRAWL_OAUTH_ISSUER", issuer]]) {
+        let u;
+        try {
+            u = new URL(v);
+        }
+        catch {
+            throw new Error(`${name} '${v}' is not an absolute URL`);
+        }
+        if (u.protocol !== "https:" && u.protocol !== "http:")
+            throw new Error(`${name} '${v}' must be http(s)`);
+    }
+    let appsChallenge = (env.SPICRAWL_OPENAI_APPS_CHALLENGE ?? "").trim();
+    const file = (env.SPICRAWL_OPENAI_APPS_CHALLENGE_FILE ?? "").trim();
+    if (!appsChallenge && file)
+        appsChallenge = readFileSync(file, "utf8").trim();
+    return {
+        enabled: isTruthy(env.SPICRAWL_CHATGPT_ENABLED),
+        resource,
+        issuer,
+        introspectSecret: (env.SPICRAWL_OAUTH_INTROSPECT_SECRET ?? "").trim(),
+        rootPrm: isTruthy(env.SPICRAWL_CHATGPT_ROOT_PRM),
+        appsChallenge,
+    };
+}
+/** Where the resource's metadata lives: `/.well-known/oauth-protected-resource` + the resource's path (RFC 9728 §3.1). */
+export function prmPath(resource) {
+    const path = new URL(resource).pathname.replace(/\/+$/, "");
+    return `/.well-known/oauth-protected-resource${path === "/" ? "" : path}`;
+}
+/** The metadata's absolute URL, as the challenge names it. */
+export function prmURL(resource) {
+    return new URL(resource).origin + prmPath(resource);
+}
+/** The protected-resource metadata document (RFC 9728). */
+export function protectedResourceMetadata(config) {
+    return {
+        resource: config.resource,
+        authorization_servers: [config.issuer],
+        scopes_supported: [...SCOPES],
+        resource_documentation: RESOURCE_DOCUMENTATION,
+    };
+}
+/** A quoted-string parameter value: no quotes, backslashes or control characters. */
+const quotable = (s) => s.replace(/["\\\u0000-\u001f\u007f]/g, " ");
+/**
+ * A `Bearer` challenge (RFC 6750 §3, RFC 9728 §5.1). Without `error` it is the
+ * plain "authenticate here" answer to a request that sent no token.
+ */
+export function bearerChallenge(resource, opts = {}) {
+    const parts = [`resource_metadata="${prmURL(resource)}"`, `scope="${quotable(opts.scope ?? SCOPES.join(" "))}"`];
+    if (opts.error)
+        parts.push(`error="${opts.error}"`);
+    if (opts.error && opts.description)
+        parts.push(`error_description="${quotable(opts.description)}"`);
+    return `Bearer ${parts.join(", ")}`;
+}
+const tokenHash = (token) => createHash("sha256").update(token).digest("hex");
+/**
+ * Introspects access tokens at `<issuer>/api/oauth/introspect`, caching
+ * positive answers by the token's SHA-256 for at most 30 s and never past the
+ * token's expiry. A negative answer is never cached, so a token issued a moment
+ * later is not refused from memory.
+ */
+export class Introspector {
+    config;
+    cache = new Map();
+    log;
+    constructor(config, options = {}) {
+        this.config = config;
+        this.log = options.log ?? (() => { });
+        if (options.now)
+            this.now = options.now;
+    }
+    now = () => Date.now();
+    get endpoint() {
+        return `${this.config.issuer}/api/oauth/introspect`;
+    }
+    /** Forgets a token, e.g. once the API refused the key it stood for. */
+    forget(token) {
+        this.cache.delete(tokenHash(token));
+    }
+    async introspect(token) {
+        const key = tokenHash(token);
+        const now = this.now();
+        const hit = this.cache.get(key);
+        if (hit) {
+            if (hit.until > now)
+                return { ok: true, grant: hit.grant };
+            this.cache.delete(key);
+        }
+        if (!this.config.introspectSecret) {
+            this.log("chatgpt: SPICRAWL_OAUTH_INTROSPECT_SECRET is not set; cannot verify access tokens");
+            return { ok: false, status: 503, description: "token verification is not configured on this server" };
+        }
+        let res;
+        try {
+            res = await fetch(this.endpoint, {
+                method: "POST",
+                headers: {
+                    Authorization: `Bearer ${this.config.introspectSecret}`,
+                    "Content-Type": "application/x-www-form-urlencoded",
+                    Accept: "application/json",
+                    "User-Agent": USER_AGENT,
+                },
+                body: new URLSearchParams({ token }).toString(),
+                signal: AbortSignal.timeout(INTROSPECT_TIMEOUT_MS),
+            });
+        }
+        catch (err) {
+            this.log("chatgpt: introspection unreachable:", err instanceof Error ? err.message : String(err));
+            return { ok: false, status: 503, description: "could not reach the authorization server to verify the token" };
+        }
+        const text = await res.text().catch(() => "");
+        if (!res.ok) {
+            // A refusal of OUR credential (401/403) or a server error says nothing about the caller's token.
+            this.log(`chatgpt: introspection answered HTTP ${res.status}`);
+            return { ok: false, status: 503, description: "the authorization server could not verify the token" };
+        }
+        let body;
+        try {
+            const parsed = JSON.parse(text);
+            if (!parsed || typeof parsed !== "object" || Array.isArray(parsed))
+                throw new Error("not an object");
+            body = parsed;
+        }
+        catch {
+            this.log("chatgpt: introspection answered a body that is not a JSON object");
+            return { ok: false, status: 503, description: "the authorization server could not verify the token" };
+        }
+        if (body.active !== true)
+            return { ok: false, status: 401, description: "The access token is invalid, expired or revoked" };
+        if (body.aud !== this.config.resource) {
+            return { ok: false, status: 401, description: "The access token was not issued for this resource" };
+        }
+        const exp = typeof body.exp === "number" && Number.isFinite(body.exp) ? body.exp * 1000 : undefined;
+        if (exp !== undefined && exp <= now)
+            return { ok: false, status: 401, description: "The access token has expired" };
+        const apiKey = typeof body.api_key === "string" ? body.api_key : "";
+        if (!KEY_RE.test(apiKey)) {
+            this.log("chatgpt: an active token came back without a usable API credential");
+            return { ok: false, status: 401, description: "The access token is not linked to a Spicrawl project" };
+        }
+        const grant = {
+            scopes: new Set(typeof body.scope === "string" ? body.scope.split(/\s+/).filter(Boolean) : []),
+            apiKey,
+            clientId: typeof body.client_id === "string" ? body.client_id : undefined,
+            expiresAt: exp,
+        };
+        const until = Math.min(now + CACHE_MAX_MS, exp ?? Infinity);
+        if (this.cache.size >= CACHE_MAX_ENTRIES)
+            this.evict(now);
+        this.cache.set(key, { grant, until });
+        return { ok: true, grant };
+    }
+    /** Drops expired entries, then the oldest ones, to stay under the cap. */
+    evict(now) {
+        for (const [k, e] of this.cache)
+            if (e.until <= now)
+                this.cache.delete(k);
+        for (const k of this.cache.keys()) {
+            if (this.cache.size < CACHE_MAX_ENTRIES)
+                break;
+            this.cache.delete(k);
+        }
+    }
+}
+//# sourceMappingURL=oauth.js.map

package/dist/chatgpt/oauth.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"oauth.js","sourceRoot":"","sources":["../../src/chatgpt/oauth.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAEH,OAAO,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACzC,OAAO,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACvC,OAAO,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAC1C,OAAO,EAAE,QAAQ,EAAE,MAAM,cAAc,CAAC;AACxC,OAAO,EAAE,MAAM,EAAE,MAAM,gBAAgB,CAAC;AAExC,4EAA4E;AAC5E,MAAM,CAAC,MAAM,MAAM,GAAG,CAAC,QAAQ,EAAE,OAAO,CAAU,CAAC;AAGnD,MAAM,CAAC,MAAM,gBAAgB,GAAG,sCAAsC,CAAC;AACvE,MAAM,CAAC,MAAM,cAAc,GAAG,0BAA0B,CAAC;AACzD,MAAM,CAAC,MAAM,sBAAsB,GAAG,sCAAsC,CAAC;AAC7E,oEAAoE;AACpE,MAAM,CAAC,MAAM,eAAe,GAAG,kCAAkC,CAAC;AAElE,MAAM,qBAAqB,GAAG,KAAK,CAAC;AACpC,kGAAkG;AAClG,MAAM,CAAC,MAAM,YAAY,GAAG,MAAM,CAAC;AACnC,MAAM,iBAAiB,GAAG,MAAM,CAAC;AAiBjC,MAAM,OAAO,GAAG,CAAC,CAAqB,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;AAEhF,mFAAmF;AACnF,MAAM,UAAU,oBAAoB,CAAC,MAAyB,OAAO,CAAC,GAAG;IACvE,MAAM,QAAQ,GAAG,OAAO,CAAC,GAAG,CAAC,yBAAyB,CAAC,IAAI,gBAAgB,CAAC;IAC5E,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,qBAAqB,CAAC,IAAI,cAAc,CAAC;IACpE,KAAK,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC,2BAA2B,EAAE,QAAQ,CAAC,EAAE,CAAC,uBAAuB,EAAE,MAAM,CAAC,CAAC,EAAE,CAAC;QACrG,IAAI,CAAM,CAAC;QACX,IAAI,CAAC;YACH,CAAC,GAAG,IAAI,GAAG,CAAC,CAAC,CAAC,CAAC;QACjB,CAAC;QAAC,MAAM,CAAC;YACP,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;QAC3D,CAAC;QACD,IAAI,CAAC,CAAC,QAAQ,KAAK,QAAQ,IAAI,CAAC,CAAC,QAAQ,KAAK,OAAO;YAAE,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,KAAK,CAAC,mBAAmB,CAAC,CAAC;IAC3G,CAAC;IACD,IAAI,aAAa,GAAG,CAAC,GAAG,CAAC,8BAA8B,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;IACtE,MAAM,IAAI,GAAG,CAAC,GAAG,CAAC,mCAAmC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;IACpE,IAAI,CAAC,aAAa,IAAI,IAAI;QAAE,aAAa,GAAG,YAAY,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,IAAI,EAAE,CAAC;IAC9E,OAAO;QACL,OAAO,EAAE,QAAQ,CAAC,GAAG,CAAC,wBAAwB,CAAC;QAC/C,QAAQ;QACR,MAAM;QACN,gBAAgB,EAAE,CAAC,GAAG,CAAC,gCAAgC,IAAI,EAAE,CAAC,CAAC,IAAI,EAAE;QACrE,OAAO,EAAE,QAAQ,CAAC,GAAG,CAAC,yBAAyB,CAAC;QAChD,aAAa;KACd,CAAC;AACJ,CAAC;AAED,0HAA0H;AAC1H,MAAM,UAAU,OAAO,CAAC,QAAgB;IACtC,MAAM,IAAI,GAAG,IAAI,GAAG,CAAC,QAAQ,CAAC,CAAC,QAAQ,CAAC,OAAO,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;IAC5D,OAAO,wCAAwC,IAAI,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;AAC5E,CAAC;AAED,8DAA8D;AAC9D,MAAM,UAAU,MAAM,CAAC,QAAgB;IACrC,OAAO,IAAI,GAAG,CAAC,QAAQ,CAAC,CAAC,MAAM,GAAG,OAAO,CAAC,QAAQ,CAAC,CAAC;AACtD,CAAC;AAED,2DAA2D;AAC3D,MAAM,UAAU,yBAAyB,CAAC,MAAqB;IAC7D,OAAO;QACL,QAAQ,EAAE,MAAM,CAAC,QAAQ;QACzB,qBAAqB,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC;QACtC,gBAAgB,EAAE,CAAC,GAAG,MAAM,CAAC;QAC7B,sBAAsB,EAAE,sBAAsB;KAC/C,CAAC;AACJ,CAAC;AAED,qFAAqF;AACrF,MAAM,QAAQ,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,2BAA2B,EAAE,GAAG,CAAC,CAAC;AAE5E;;;GAGG;AACH,MAAM,UAAU,eAAe,CAC7B,QAAgB,EAChB,OAAiG,EAAE;IAEnG,MAAM,KAAK,GAAG,CAAC,sBAAsB,MAAM,CAAC,QAAQ,CAAC,GAAG,EAAE,UAAU,QAAQ,CAAC,IAAI,CAAC,KAAK,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC;IACjH,IAAI,IAAI,CAAC,KAAK;QAAE,KAAK,CAAC,IAAI,CAAC,UAAU,IAAI,CAAC,KAAK,GAAG,CAAC,CAAC;IACpD,IAAI,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,WAAW;QAAE,KAAK,CAAC,IAAI,CAAC,sBAAsB,QAAQ,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC;IACpG,OAAO,UAAU,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;AACtC,CAAC;AAsBD,MAAM,SAAS,GAAG,CAAC,KAAa,EAAE,EAAE,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;AAEtF;;;;;GAKG;AACH,MAAM,OAAO,YAAY;IAKJ;IAJF,KAAK,GAAG,IAAI,GAAG,EAAsB,CAAC;IACtC,GAAG,CAA4B;IAEhD,YACmB,MAAuE,EACxF,UAAmE,EAAE;QADpD,WAAM,GAAN,MAAM,CAAiE;QAGxF,IAAI,CAAC,GAAG,GAAG,OAAO,CAAC,GAAG,IAAI,CAAC,GAAG,EAAE,GAAE,CAAC,CAAC,CAAC;QACrC,IAAI,OAAO,CAAC,GAAG;YAAE,IAAI,CAAC,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC;IAC1C,CAAC;IAEO,GAAG,GAAG,GAAG,EAAE,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC;IAE/B,IAAI,QAAQ;QACV,OAAO,GAAG,IAAI,CAAC,MAAM,CAAC,MAAM,uBAAuB,CAAC;IACtD,CAAC;IAED,uEAAuE;IACvE,MAAM,CAAC,KAAa;QAClB,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,CAAC,CAAC;IACtC,CAAC;IAED,KAAK,CAAC,UAAU,CAAC,KAAa;QAC5B,MAAM,GAAG,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC;QAC7B,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QAChC,IAAI,GAAG,EAAE,CAAC;YACR,IAAI,GAAG,CAAC,KAAK,GAAG,GAAG;gBAAE,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,CAAC;YAC3D,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACzB,CAAC;QAED,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,gBAAgB,EAAE,CAAC;YAClC,IAAI,CAAC,GAAG,CAAC,mFAAmF,CAAC,CAAC;YAC9F,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,qDAAqD,EAAE,CAAC;QACxG,CAAC;QAED,IAAI,GAAa,CAAC;QAClB,IAAI,CAAC;YACH,GAAG,GAAG,MAAM,KAAK,CAAC,IAAI,CAAC,QAAQ,EAAE;gBAC/B,MAAM,EAAE,MAAM;gBACd,OAAO,EAAE;oBACP,aAAa,EAAE,UAAU,IAAI,CAAC,MAAM,CAAC,gBAAgB,EAAE;oBACvD,cAAc,EAAE,mCAAmC;oBACnD,MAAM,EAAE,kBAAkB;oBAC1B,YAAY,EAAE,UAAU;iBACzB;gBACD,IAAI,EAAE,IAAI,eAAe,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,QAAQ,EAAE;gBAC/C,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,qBAAqB,CAAC;aACnD,CAAC,CAAC;QACL,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,IAAI,CAAC,GAAG,CAAC,qCAAqC,EAAE,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;YAClG,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,8DAA8D,EAAE,CAAC;QACjH,CAAC;QACD,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC,EAAE,CAAC,CAAC;QAC9C,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;YACZ,iGAAiG;YACjG,IAAI,CAAC,GAAG,CAAC,wCAAwC,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC;YAC/D,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,qDAAqD,EAAE,CAAC;QACxG,CAAC;QACD,IAAI,IAA6B,CAAC;QAClC,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAY,CAAC;YAC3C,IAAI,CAAC,MAAM,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC;gBAAE,MAAM,IAAI,KAAK,CAAC,eAAe,CAAC,CAAC;YACrG,IAAI,GAAG,MAAiC,CAAC;QAC3C,CAAC;QAAC,MAAM,CAAC;YACP,IAAI,CAAC,GAAG,CAAC,kEAAkE,CAAC,CAAC;YAC7E,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,qDAAqD,EAAE,CAAC;QACxG,CAAC;QAED,IAAI,IAAI,CAAC,MAAM,KAAK,IAAI;YAAE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,iDAAiD,EAAE,CAAC;QAC5H,IAAI,IAAI,CAAC,GAAG,KAAK,IAAI,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC;YACtC,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,mDAAmD,EAAE,CAAC;QACtG,CAAC;QACD,MAAM,GAAG,GAAG,OAAO,IAAI,CAAC,GAAG,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC;QACpG,IAAI,GAAG,KAAK,SAAS,IAAI,GAAG,IAAI,GAAG;YAAE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,8BAA8B,EAAE,CAAC;QACpH,MAAM,MAAM,GAAG,OAAO,IAAI,CAAC,OAAO,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;QACpE,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC;YACzB,IAAI,CAAC,GAAG,CAAC,oEAAoE,CAAC,CAAC;YAC/E,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,EAAE,WAAW,EAAE,sDAAsD,EAAE,CAAC;QACzG,CAAC;QAED,MAAM,KAAK,GAAU;YACnB,MAAM,EAAE,IAAI,GAAG,CAAC,OAAO,IAAI,CAAC,KAAK,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC;YAC9F,MAAM;YACN,QAAQ,EAAE,OAAO,IAAI,CAAC,SAAS,KAAK,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,SAAS;YACzE,SAAS,EAAE,GAAG;SACf,CAAC;QACF,MAAM,KAAK,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,GAAG,YAAY,EAAE,GAAG,IAAI,QAAQ,CAAC,CAAC;QAC5D,IAAI,IAAI,CAAC,KAAK,CAAC,IAAI,IAAI,iBAAiB;YAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;QAC1D,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,CAAC,CAAC;QACtC,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC;IAC7B,CAAC;IAED,0EAA0E;IAClE,KAAK,CAAC,GAAW;QACvB,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,KAAK;YAAE,IAAI,CAAC,CAAC,KAAK,IAAI,GAAG;gBAAE,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;QAC1E,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,KAAK,CAAC,IAAI,EAAE,EAAE,CAAC;YAClC,IAAI,IAAI,CAAC,KAAK,CAAC,IAAI,GAAG,iBAAiB;gBAAE,MAAM;YAC/C,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;QACvB,CAAC;IACH,CAAC;CACF"}

package/dist/chatgpt/tools.d.ts

@@ -0,0 +1,45 @@
+/**
+ * The restricted tool profile served on `/chatgpt/mcp`.
+ *
+ * A smaller, read-mostly set than `/mcp`, shaped by OpenAI's app guidelines:
+ * a scrape is always a GET with no browser actions, custom headers, proxies,
+ * sessions or engine pins; a batch takes a short list of URLs with a credit
+ * budget; results come back without request ids, timestamps or internal ids.
+ * Every tool states its OAuth scopes (`securitySchemes`, top level and in
+ * `_meta`) so the client can ask for the right consent.
+ */
+import { ILayerClient } from "../client.js";
+import { SpicrawlMcpServer } from "../server.js";
+import type { Scope } from "./oauth.js";
+export type SecurityScheme = {
+    type: "oauth2";
+    scopes: Scope[];
+};
+/** Each tool on this surface with the scopes a call to it needs. Anything else registered is dropped. */
+export declare const TOOL_SCOPES: Record<string, Scope[]>;
+export declare function sanitizeScrape(data: unknown): unknown;
+export declare function sanitizeJob(data: unknown): unknown;
+export declare function sanitizeResults(data: unknown): unknown;
+export interface ChatgptContext {
+    /**
+     * The WWW-Authenticate challenge for a tool error when the API refused the
+     * credential behind the token mid-call (`_meta["mcp/www_authenticate"]`).
+     */
+    upstreamChallenge: (scopes: Scope[]) => string;
+    /** Called when the API refused that credential, so the token is not trusted from cache. */
+    onUpstreamUnauthorized: () => void;
+}
+/**
+ * The results endpoint inlines `result.content` only once the whole job is
+ * terminal (it reads the stitched result file), but items report `succeeded`
+ * as each finishes. A model that sees every item succeeded and reads results
+ * before the job turns `completed` would get no content at all, so a succeeded
+ * item without a `result` is filled from GET /v1/batch/{id}/tasks/{seq}/content,
+ * which serves a finished item's payload on its own. An item it cannot serve
+ * is marked `unavailable` instead of failing the page, and past FILL_MAX_BYTES
+ * an item is marked `omitted`. A 401 still propagates,
+ * so `call` can answer with a fresh OAuth challenge.
+ */
+export declare function fillMissingContent(client: ILayerClient, job_id: string, page: Record<string, unknown>): Promise<void>;
+/** One MCP server for one ChatGPT request, running under the API key the token stands for. */
+export declare function buildChatgptServer(client: ILayerClient, ctx: ChatgptContext): SpicrawlMcpServer;