mcp-scraper 0.1.9 → 0.2.0

package/dist/{worker-AUCXFHEL.js → worker-KJ4A7WIR.js}

@@ -4,7 +4,7 @@ import {
   createHarvestAttemptRecorder,
   harvestProblemResponse,
   serializeHarvestProblem
-} from "./chunk-ZK456YXN.js";
+} from "./chunk-GXBT5CDU.js";
 import {
   browserServiceApiKey,
   harvest
@@ -122,4 +122,4 @@ export {
   startWorker,
   tickOnce
 };
-//# sourceMappingURL=worker-AUCXFHEL.js.map
+//# sourceMappingURL=worker-KJ4A7WIR.js.map

package/docs/specs/api-forge-spec.md

@@ -0,0 +1,234 @@
+# API Forge — Implementation Spec
+
+Research-grounded API design intelligence inside MCP Scraper. A database-design tool asks about entities and relationships, then emits DDL. API Forge asks about resources, operations, and consumers — then emits OpenAPI, Zod schemas, SQL DDL, and **MCP tool definitions that comply with `docs/mcp-tool-quality-spec.md`** — with every design decision groundable in live web evidence harvested by the existing scraper tools.
+
+## The three-party design
+
+This is the core architecture decision and the creative center of the feature:
+
+| Party | Owns | Never does |
+|---|---|---|
+| **MCP Scraper server** | Methodology (phase machine, question cards), session state, research execution, artifact generation, critique rules | Conversation with the human |
+| **Calling LLM** (Claude, etc.) | Interviewing the human, translating answers into structured form, narrating findings | Design methodology, state |
+| **Scraper tools** (existing) | Evidence: PAA questions developers ask, competitor API doc conventions, domain entity discovery | — |
+
+The server returns *question cards* and *synthesis instructions*; the calling model conducts the interview and reports structured answers back. This works on every MCP host (no dependency on elicitation support) and keeps the methodology versioned server-side.
+
+## Tool surface (5 new MCP tools)
+
+All registered in `src/mcp/paa-mcp-server.ts`, schemas in `src/mcp/mcp-tool-schemas.ts`, formatters in `src/mcp/mcp-response-formatter.ts`. All have `outputSchema` + `structuredContent` (these are chaining tools by definition). Annotations: `api_critique` is `readOnlyHint: true`; the four session tools are `readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true` (they mutate server-side session state).
+
+### 1. `api_design_start`
+
+```
+inputSchema: {
+  name:    z.string().min(1).describe('Working name of the API, e.g. "Acme Bookings API"'),
+  purpose: z.string().min(1).describe('One-sentence purpose in the user\'s words'),
+  domain:  z.string().min(1).describe('Business domain, e.g. "salon appointment booking" — used to seed research'),
+  research: z.boolean().default(true).describe('Run an automatic domain research pass (PAA + SERP) before the first interview card. Costs credits per the cost table.'),
+}
+```
+
+Behavior: creates a session, optionally runs Phase-0 research (internal calls to `harvest()` with `maxQuestions: 15` and query `"${domain} api"` — direct function call, not HTTP; billed via existing `paa` ledger op), and returns the first question card.
+
+Description (model-facing, per quality spec): "Start an API design session. Returns a sessionId, optional domain research findings, and the first interview card — a batch of design questions. Interview the user conversationally using the card, then submit their answers with api_design_answer. Do not invent answers the user did not give."
+
+`outputSchema`: `{ sessionId, phase, card: CardOutput, research: { paaQuestions: string[], topCompetitorUrls: string[] } | null }`
+
+### 2. `api_design_answer`
+
+```
+inputSchema: {
+  sessionId: z.string().min(1),
+  answers:   z.array(z.object({
+    questionId: z.string().min(1),
+    value:      z.string().min(1).describe('The user\'s answer, verbatim or faithfully summarized'),
+    skipped:    z.boolean().default(false),
+  })).min(1),
+}
+```
+
+Behavior: validates answers against the current card, folds them into the `DesignModel` via per-question reducers, advances the phase machine when the card is satisfied, returns the next card (or `phase: 'complete'`). Free (no debit) — answers are state writes only.
+
+`outputSchema`: `{ sessionId, phase, accepted: number, designSummary: DesignSummaryOutput, card: CardOutput | null, complete: boolean }`
+
+### 3. `api_design_research`
+
+```
+inputSchema: {
+  sessionId:      z.string().min(1),
+  competitorUrls: z.array(z.string().url()).max(5).optional().describe('API documentation URLs to mine for conventions (pagination idiom, error shape, naming, auth)'),
+  topic:          z.string().optional().describe('Free-form research topic, e.g. "webhook retry best practices" — runs a PAA harvest'),
+}
+```
+
+Behavior: for each competitor URL, runs the existing `extract_url` pipeline internally, then a **deterministic convention extractor** (no server-side LLM): regex/heading analysis of the extracted Markdown detecting pagination style (`cursor|page|offset` token frequency), error envelope (`"error"`/`"errors"`/problem+json mentions), naming case (snake vs camel ratio in code blocks), auth scheme mentions. For `topic`, runs a PAA harvest. Results are stored as `evidence[]` entries on the session and returned with a `synthesisInstruction` string telling the calling model how to present findings and which open design questions the evidence bears on. Billed: `page_scrape` per URL, `paa` per question — existing ledger ops, no new rates needed.
+
+### 4. `api_design_blueprint`
+
+```
+inputSchema: {
+  sessionId: z.string().min(1),
+  formats:   z.array(z.enum(['openapi', 'zod', 'sql', 'mcp_tools', 'readme'])).min(1)
+               .default(['openapi', 'zod', 'sql', 'mcp_tools', 'readme']),
+}
+```
+
+Behavior: runs the generators (below) against the materialized `DesignModel`. Allowed mid-interview (partial blueprint from whatever is designed so far — this enables the "show me where we are" loop). Returns artifacts inline as fenced code blocks in `content` plus `structuredContent.artifacts: [{ format, filename, source }]`. On stdio, each artifact is also saved through the existing `saveFullReport` path, which makes blueprints appear in the `report://` resources list for free. New rate: `api_blueprint` 1 000 mc (1 credit) per generation.
+
+### 5. `api_critique`
+
+```
+inputSchema: {
+  openapiUrl: z.string().url().optional().describe('URL of an existing OpenAPI/Swagger document, or API docs page'),
+  openapiSource: z.string().optional().describe('Pasted OpenAPI YAML/JSON when no URL is available'),
+  audience:   z.enum(['human', 'service', 'ai_agent']).default('ai_agent').describe('Score the API for this consumer. ai_agent applies MCP-readiness rules.'),
+}
+```
+
+Behavior: fetch (via internal `extract_url` when URL) → parse OpenAPI → run the critique rule table → return scored findings `[{ ruleId, severity: 'P0'|'P1'|'P2', location, finding, fix }]`. The `ai_agent` audience adds the MCP-readiness rules derived from `docs/mcp-tool-quality-spec.md` (description quality, enum documentation, error actionability, schema-encoded defaults/caps). This is the standalone marketing wedge: "run our quality spec against *your* API." New rate: `api_critique` 2 000 mc.
+
+## Reverse Forge (the second creative wedge — Phase 6, optional)
+
+`api_design_start` accepts an optional `importUrl`: point it at an existing API's docs site; the server runs `map_site_urls` + `extract_url` over the docs internally, reconstructs a partial `DesignModel` from the conventions extractor, and starts the interview at the *gaps* instead of from zero. The session then supports "extend this API with X" with consistency checks against the inferred conventions (e.g. "your existing endpoints use cursor pagination and snake_case; the new webhooks resource will follow"). Implemented as `meta.importedFrom` on the session plus a `seedDesignFromDocs(extracts: ExtractResult[]): Partial<DesignModel>` function in the conventions extractor module.
+
+## The interview engine
+
+### `src/forge/design-model.ts` — exact types
+
+```ts
+export interface DesignField { name: string; type: 'string'|'number'|'boolean'|'timestamp'|'json'|'ref'; refEntity?: string; required: boolean; description: string }
+export interface DesignEntity { name: string; plural: string; identity: 'uuid'|'slug'|'int'; fields: DesignField[] }
+export interface DesignRelationship { from: string; to: string; kind: 'one_to_one'|'one_to_many'|'many_to_many'; ownership: 'embedded'|'referenced' }
+export interface DesignOperation { entity: string; verb: 'list'|'get'|'create'|'update'|'delete'|'action'; actionName?: string; idempotent: boolean; paginated: boolean; description: string }
+export interface DesignContracts { errorShape: 'problem_json'|'envelope'|'custom'; pagination: 'cursor'|'offset'|'none'; naming: 'snake'|'camel'; envelope: boolean }
+export interface DesignAuth { scheme: 'api_key'|'oauth2'|'jwt'|'none'; scopes: string[] }
+export interface DesignEvidence { source: 'paa'|'serp'|'competitor_doc'|'user'; url: string | null; finding: string; appliedTo: string }
+export interface DesignModel {
+  meta: { name: string; purpose: string; domain: string; consumers: Array<'human'|'service'|'ai_agent'>; importedFrom?: string }
+  entities: DesignEntity[]
+  relationships: DesignRelationship[]
+  operations: DesignOperation[]
+  contracts: DesignContracts
+  auth: DesignAuth
+  evidence: DesignEvidence[]
+}
+```
+
+### `src/forge/question-cards.ts` — phase machine as data
+
+```ts
+export type ForgePhase = 'consumers'|'entities'|'relationships'|'operations'|'contracts'|'auth'|'review'|'complete'
+export interface QuestionCard {
+  phase: ForgePhase
+  intro: string                     // one paragraph the model relays before asking
+  questions: Array<{
+    id: string                      // e.g. 'entities.list'
+    ask: string                     // the question, written to be read aloud
+    why: string                     // rationale the model can offer if asked
+    expects: 'free_text'|'entity_list'|'choice'|'yes_no'
+    choices?: string[]
+    skippable: boolean
+    appliesIf?: (model: DesignModel) => boolean   // skip logic, e.g. m2m junction question only if any many_to_many
+  }>
+}
+export const CARDS: Record<ForgePhase, QuestionCard>
+```
+
+Phase order and headline questions (full card text lives in the file):
+
+1. **consumers** — "Who calls this API: humans via a UI, backend services, or AI agents?" (multi-choice; `ai_agent` switches on MCP generation + agent-readability rules). "What must never be exposed?" (seeds redaction rules — mirrors this repo's vendor-concealment pattern).
+2. **entities** — "Name the things your API manages, singular nouns." Then per entity: identity strategy, 3–7 core fields. Reducer: `entities.list` answer is parsed as comma/newline-separated nouns; per-entity follow-up cards are generated dynamically (`entities.fields.{name}`).
+3. **relationships** — for each entity pair that co-occurred in answers: "Does an X own many Y?" Cardinality + embedded-vs-referenced.
+4. **operations** — per entity: which of list/get/create/update/delete, plus domain actions ("cancel a booking" → `POST /bookings/{id}/cancel`, `idempotent: false`).
+5. **contracts** — pagination, error shape, naming. **This is where research lands**: if the session has competitor evidence, the card's `intro` includes "Competitors X and Y both use cursor pagination" and the question defaults accordingly.
+6. **auth** — scheme + scopes; if consumers includes `ai_agent`, ask about per-call cost/metering (seeds a credits-style design like this repo's).
+7. **review** — the server returns a `DesignSummary`; the model walks the user through it; any "change X" answers route back to the owning phase.
+
+### `src/forge/interview-engine.ts`
+
+```ts
+export function nextCard(model: DesignModel, phase: ForgePhase): { card: QuestionCard | null; phase: ForgePhase }
+export function applyAnswers(model: DesignModel, phase: ForgePhase, answers: Answer[]): { model: DesignModel; errors: string[] }
+export function designSummary(model: DesignModel): DesignSummaryOutput
+```
+
+Pure functions — no I/O — so the whole engine is unit-testable with golden sessions.
+
+## Generators — `src/forge/generators/`
+
+Each is `(model: DesignModel) => { filename: string; source: string }`, pure, individually golden-tested.
+
+- `openapi.ts` — OpenAPI 3.1 YAML. Paths from operations (`GET /bookings`, `POST /bookings/{id}/cancel`), components.schemas from entities, error responses per `contracts.errorShape`, pagination params per `contracts.pagination`, securitySchemes per auth.
+- `zod.ts` — one Zod schema per entity + per-operation input schemas, naming per `contracts.naming`.
+- `sql.ts` — SQLite-flavored DDL: one table per entity, junction tables for `many_to_many`, FK columns for `referenced` one-to-many, `TEXT` ISO timestamps (house style of this repo's `db.ts`).
+- `mcp-tools.ts` — the dogfood crown: emits a `registerTool` block per operation following `docs/mcp-tool-quality-spec.md` — model-instructing description ("Use this when…"), schema-encoded defaults/caps, annotations (`readOnlyHint: true` for list/get), `outputSchema` for chaining ops, error-shape guidance. Generated against the same `liveWebToolAnnotations` helper pattern used in `paa-mcp-server.ts`.
+- `readme.ts` — install + auth + per-endpoint docs with curl examples.
+
+## Persistence — `src/api/db.ts` additions
+
+```sql
+CREATE TABLE IF NOT EXISTS api_forge_sessions (
+  id          TEXT PRIMARY KEY,
+  user_id     INTEGER NOT NULL,
+  name        TEXT NOT NULL,
+  phase       TEXT NOT NULL DEFAULT 'consumers',
+  status      TEXT NOT NULL DEFAULT 'active',
+  design_json TEXT NOT NULL DEFAULT '{}',
+  created_at  TEXT NOT NULL,
+  updated_at  TEXT NOT NULL
+);
+CREATE TABLE IF NOT EXISTS api_forge_events (
+  id           INTEGER PRIMARY KEY AUTOINCREMENT,
+  session_id   TEXT NOT NULL,
+  kind         TEXT NOT NULL,        -- 'answers' | 'research' | 'blueprint'
+  payload_json TEXT NOT NULL,
+  created_at   TEXT NOT NULL
+);
+```
+
+`design_json` is the materialized DesignModel; events are the audit trail (enables replay/undo later). DB helpers: `createForgeSession`, `getForgeSession`, `updateForgeSession`, `appendForgeEvent` — same row-mapping style as `getJob`.
+
+## Routes — `src/api/api-forge-routes.ts`
+
+`forgeApp = new Hono<ApiKeyEnv>()` mounted at `/api-forge` in `server.ts` and added to `vercel.json` rewrites (`{ "source": "/api-forge/:path*", "destination": "/api/index" }`). Five POST routes mirroring the tools; `createApiKeyAuth`; debit-before-work + refund-on-failure exactly like `maps-routes.ts`; errors in the structured `{error, error_code, retryable}` shape. Session ownership check on every route (`session.user_id === user.id` else 404).
+
+## Rates — `src/api/rates.ts` additions
+
+```ts
+api_forge_start: 2_000,   // 2 credits, includes Phase-0 research overhead
+api_blueprint:   1_000,
+api_critique:    2_000,
+```
+
+Plus `CREDIT_COST_CATALOG` entries with aliases (`'api design'`, `'blueprint'`, `'critique'`), `LedgerOperation.API_FORGE_START / API_BLUEPRINT / API_CRITIQUE` + paired `_REFUND` ops (the `ledger-refund-keys` test enforces pairing).
+
+## MCP wiring
+
+- Executor: 5 methods on `IMcpToolExecutor`… **no** — these are a separate concern; add `IApiForgeToolExecutor` interface in `IMcpToolExecutor.ts` (same pattern as `ISerpIntelligenceToolExecutor`), implemented by `HttpMcpToolExecutor` (`this.call('/api-forge/start', input)` etc.).
+- Registration: `registerApiForgeTools(server, executor)` in `paa-mcp-server.ts`, called from both transports (these tools work identically hosted and stdio — session state is server-side).
+- Formatters: each returns a Markdown report (`oneBlock`) + `structuredContent`. The blueprint formatter renders each artifact as a fenced block and saves per-artifact files on stdio.
+
+## Tests
+
+- `tests/unit/forge-interview-engine.test.ts` — phase transitions, skip logic, reducer correctness, a full golden session (fixture answers → expected DesignModel).
+- `tests/unit/forge-generators.test.ts` — golden outputs per generator from a fixture DesignModel; the `mcp-tools` generator output is itself grep-asserted for quality-spec markers (`readOnlyHint`, `.default(`, `.max(`).
+- `tests/unit/forge-conventions-extractor.test.ts` — pagination/error/naming detection from fixture doc Markdown.
+- `tests/unit/mcp-server-registration.test.ts` — tool count 13 → 18 (stdio) / 20 (hosted); annotations table updated.
+- `tests/live/mcp/api-forge.live.test.ts` — start → answer × 2 → blueprint over stdio against the test server; asserts artifacts parse (YAML loads, DDL contains `CREATE TABLE`).
+
+## Build phases
+
+1. **Engine** — design-model.ts, question-cards.ts, interview-engine.ts + unit tests. No I/O. (Largest thinking, smallest risk.)
+2. **Persistence + routes** — db.ts tables, api-forge-routes.ts (start/answer only), rates, vercel.json rewrite.
+3. **Generators** — all five + golden tests; blueprint route + tool.
+4. **Research** — conventions extractor, internal harvest/extract calls, evidence folding, research tool.
+5. **Critique** — rule table + tool (independent of sessions; can ship before 4 if desired).
+6. **Reverse Forge** — importUrl seeding (optional, after 1–5 prove out).
+7. **Release** — quality-spec Definition-of-Done sweep: README, `public/skills/mcp-scraper/skill.md` tool list, registration tests, live tests, version bump, vercel deploy then npm publish.
+
+## Open decisions (resolve before Phase 2)
+
+1. Session TTL/limits: cap active sessions per user (proposal: 10) and expire after 30 days (cron at `/cron/tick` already exists).
+2. Should `api_design_answer` bill micro-credits to deter abuse, or stay free? (Proposal: free; starts are the gate.)
+3. Server-side LLM synthesis: deliberately excluded — the calling model synthesizes, the server stays deterministic. Revisit only if convention extraction proves too weak on real competitor docs.

package/docs/specs/deferred-work-spec.md

@@ -0,0 +1,74 @@
+# Deferred Work Spec
+
+Items deliberately deferred from the June 2026 release train (0.1.7 → 0.1.9), specified to execution detail. Each section is independently shippable. Ordered by recommended priority.
+
+## 1. Ops: finish the vendor env var rename (no release needed)
+
+Code already reads `BROWSER_SERVICE_API_KEY` / `BROWSER_SERVICE_PROXY_ID` first and falls back to `KERNEL_API_KEY` / `KERNEL_PROXY_ID` (`src/lib/browser-service-env.ts`). Remaining steps are pure ops:
+
+1. Local: add to `.env`: `BROWSER_SERVICE_API_KEY=<same value as KERNEL_API_KEY>` and `BROWSER_SERVICE_PROXY_ID=<same as KERNEL_PROXY_ID>`.
+2. Vercel: `vercel env add BROWSER_SERVICE_API_KEY production` (paste value), same for `BROWSER_SERVICE_PROXY_ID`; redeploy (`vercel --prod --yes`).
+3. Verify: `curl -s https://mcpscraper.dev/me -H "x-api-key: $KEY"` still 200 and a `harvest_paa` smoke call succeeds.
+4. Only after a full green release cycle: remove `KERNEL_API_KEY`/`KERNEL_PROXY_ID` from Vercel and `.env`, then delete the fallback reads in `src/lib/browser-service-env.ts:2` and `:7` and the `?? process.env.KERNEL_API_KEY` in `src/api/env.ts:35`.
+5. Update live test requirements at the same time as step 4: `tests/live/mcp/mcp-stdio-protocol.live.test.ts:23` and `tests/live/mcp/mcp-http-protocol.live.test.ts:9` — change `requireLiveEnv([... 'KERNEL_API_KEY'])` to `'BROWSER_SERVICE_API_KEY'`.
+
+## 2. Ops: repair the git remote
+
+`git push` fails with "Repository not found"; release commits 0.1.6–0.1.9 exist only locally (single point of failure on this machine).
+
+1. Decide: fix the existing `origin` URL (`git remote -v`, then `git remote set-url origin <correct-url>`) or create a fresh private repo (`gh repo create vilovieta/mcp-scraper --private --source=. --push`).
+2. Push `main` and verify all four release commits land.
+3. Update the `release-workflow` memory note once pushes work.
+
+## 3. Hosted deep-harvest progress (deferred from task #7)
+
+**Problem:** `harvest_paa` calls budgeted up to 280 s return nothing until completion. The hosted transport runs stateless JSON mode (`enableJsonResponse: true`, `mcp-routes.ts`), which cannot stream `notifications/progress`.
+
+**Design — poll-based progress via the existing async REST API (works on BOTH transports):**
+
+1. `src/mcp/http-mcp-tool-executor.ts` — new private method `callWithPolling(input, onProgress)`:
+   - `POST /harvest` (existing async endpoint, returns `202 { job_id }`).
+   - Poll `GET /jobs/{job_id}` every 5 s up to the existing tiered budget from `harvestTimeoutBudget`.
+   - Each poll, derive progress from the job's `attempts` array (already returned): `onProgress({ progress: attempts.at(-1)?.question_count ?? 0, total: input.maxQuestions })`.
+   - On `status: 'done'` return the result in the existing `CallToolResult` shape; on `failed`/`cancelled` return the structured error (shape already produced by `harvestProblemResponse`).
+2. `src/mcp/paa-mcp-server.ts` — the `harvest_paa` handler gains the SDK's progress plumbing: the third callback argument `extra` exposes `sendNotification` and `_meta.progressToken`; when a token is present, forward `onProgress` events as `notifications/progress { progressToken, progress, total }`.
+3. stdio transport: works as-is (long-lived connection).
+4. Hosted transport: flip `mcp-routes.ts` to streaming mode — `enableJsonResponse: false` — so responses go out as SSE and interleaved progress notifications are legal. Keep `sessionIdGenerator: undefined` (still stateless per-request). Verify Vercel function streaming passes SSE through `api/index.ts`'s `streamResponse` (it already streams body chunks).
+5. Gate: only use polling mode when `maxQuestions > 50`; smaller calls keep the current single-shot `/harvest/sync` path.
+6. Tests: unit — fake fetch sequencing 202 → running → done, assert progress callbacks; live — stdio harvest with `maxQuestions: 60` asserting ≥1 progress notification received by the harness (add `onNotification` capture to `tests/live/harness/mcp-protocol-client.ts`).
+
+**Estimated blast radius:** executor, server builder, mcp-routes, harness, 2 test files. No REST API changes.
+
+## 4. Full OAuth 2.1 on hosted /mcp (deferred from task #9)
+
+**Superseded by `docs/specs/oauth-mcp-spec.md`** — decision locked (WorkOS AuthKit, Clerk as documented alternate, self-hosted out of scope), full implementation spec with files, functions, env vars, tests, and rollout phases. The decision matrix below is retained for historical context.
+
+Shipped already: `WWW-Authenticate: Bearer` on 401 (`mcp-routes.ts:mcpAuthError`). Remaining work requires a **product decision** before any code: which authorization server?
+
+| Option | Effort | Notes |
+|---|---|---|
+| WorkOS / Auth0 in front | Low | Fastest path to claude.ai connector compatibility; monthly cost |
+| Self-rolled AS on Turso (authorization codes + PKCE tables) | High | No vendor; significant security surface — not recommended |
+| Stay API-key only | Zero | Forecloses web-connector onboarding; revisit when demand exists |
+
+When a decision lands, the implementation is:
+
+1. `GET /.well-known/oauth-protected-resource` (RFC 9728) served from `server.ts`: `{ resource: "https://mcpscraper.dev/mcp", authorization_servers: ["<AS issuer>"], bearer_methods_supported: ["header"] }`. Add a `vercel.json` rewrite for the path.
+2. `src/mcp/mcp-routes.ts` `requireMcpCallerKey`: accept a Bearer JWT in addition to API keys — validate signature against the AS JWKS (cache keys 1 h), require `aud` containing `https://mcpscraper.dev/mcp` (RFC 8707 resource binding), map the token's subject to a user row via a new `users.oauth_subject` column (migration in `db.ts`).
+3. Keep `x-api-key` working forever (NPX clients, REST customers).
+4. Tests: unit for token validation paths (expired/wrong-aud/garbage); live test gated on AS test-client credentials in env.
+
+## 5. outputSchema for the capture tools (deliberately excluded)
+
+`capture_serp_snapshot` / `capture_serp_page_snapshots` return raw passthrough of the SERP Intelligence product payload, which is still evolving with Phoenix. Adding `outputSchema` now would either freeze that payload or break on every product iteration. Revisit when the Phoenix capture contract stabilizes; at that point: schema in `mcp-tool-schemas.ts`, a formatter (currently none — executor result passes straight through), and registration updates in `mcp-routes.ts:registerSerpIntelligenceCaptureTools`.
+
+## 6. Maintenance follow-ups (small, batch into any release)
+
+1. `public/skill.md` / `public/codex-skill.md` (REST-mode skill files): regenerate the `maxQuestions` guidance to say cap 200 (currently 1–150 in the parsing table) — `public/skill.md` PAA Harvest Mode STEP 1 table row `maxQuestions`.
+2. `src/cli.ts:21` still names the flag `--kernel-api-key` (help text already neutral). Renaming the flag breaks invocations; add an alias `--browser-api-key` and hide the old one when commander supports it.
+3. `tests/live/harness/mcp-protocol-client.ts` `close()` uses `setTimeout(resolve, 3_000)` without unref — harmless in vitest, but unref it if the harness ever runs outside vitest.
+4. Internal naming (`BrowserDriver` kernel fields, DB columns `kernel_session_id`, `src/api/kernel-fetch.ts`, `src/kernel-proxy-resolver.ts`) intentionally unchanged — outbound sanitizers cover the boundary. Do NOT rename DB columns without a migration plan; the sanitizers make it unnecessary.
+
+## 7. API Forge
+
+Specified separately in `docs/specs/api-forge-spec.md`. Recommended to schedule after §1 and §2 (ops hygiene first), before §3/§4 (it produces revenue surface; progress/OAuth polish can follow demand).

package/docs/specs/oauth-mcp-spec.md

@@ -0,0 +1,213 @@
+# OAuth 2.1 for Hosted /mcp — Implementation Spec
+
+Lets MCP clients that support OAuth (claude.ai web connectors, future hosts) connect to `https://mcpscraper.dev/mcp` with a "Connect → log in → approve" flow instead of a pasted API key. API keys keep working forever; OAuth is additive.
+
+## Decision record
+
+**Authorization Server: WorkOS AuthKit.** Rationale: explicit MCP authorization support including Dynamic Client Registration (the piece claude.ai requires), PKCE, JWKS-signed JWT access tokens, "Sign in with Google" as a login method, free tier covering early volume. Documented alternate: Clerk (§10). Self-hosted AS: deliberately out of scope (§11).
+
+**Architecture rule that keeps the vendor swappable:** mcpscraper.dev is only the OAuth *protected resource*. All AS-specific behavior is isolated behind `src/api/oauth/` and three env vars. Swapping AS later = change env vars, re-test, no route changes.
+
+**Two non-negotiable security rules:**
+1. **No token passthrough.** The user's OAuth token is never forwarded anywhere. Internal REST calls from the MCP executor continue to use the resolved user's own `api_key` (the `users` row already carries it).
+2. **Resource binding.** Tokens are accepted only when `aud` contains the canonical resource URI `https://mcpscraper.dev/mcp` (RFC 8707). A token minted for any other API is rejected even if the same AS signed it.
+
+## The flow being implemented
+
+```
+claude.ai                      mcpscraper.dev                    WorkOS AuthKit
+   │ POST /mcp (no token)           │                                  │
+   │←─ 401 + WWW-Authenticate ──────│                                  │
+   │   resource_metadata=…          │                                  │
+   │ GET /.well-known/oauth-protected-resource ──→ JSON: AS issuer     │
+   │ GET {issuer}/.well-known/oauth-authorization-server ─────────────→│
+   │ POST {issuer}/oauth2/register  (Dynamic Client Registration) ────→│
+   │ browser: /oauth2/authorize + PKCE + resource=…/mcp  (user logs in,│
+   │          via Google or email, approves) ─────────────────────────→│
+   │ POST {issuer}/oauth2/token ──────────────────────────────────────→│
+   │ POST /mcp  Authorization: Bearer <JWT> ──→ verify JWKS + aud      │
+   │                                 └→ resolve/provision user → tools │
+```
+
+## 1. New env vars — `src/api/env.ts` + `.env` + Vercel
+
+```
+OAUTH_ENABLED=true                                  # master gate; absent/false = current behavior
+OAUTH_ISSUER=https://<subdomain>.authkit.app        # from WorkOS dashboard
+OAUTH_JWKS_URL=                                     # optional; defaults to ${OAUTH_ISSUER}/oauth2/jwks
+OAUTH_AUDIENCE=https://mcpscraper.dev/mcp           # canonical resource URI
+```
+
+`env.ts`: add all four to `RequiredEnvSchema` as `z.string().optional()`. Do **not** add to `REQUIRED_VARS` (OAuth is optional infrastructure).
+
+## 2. New dependency
+
+`npm install jose` (runtime dep, not dev). Used for `createRemoteJWKSet` + `jwtVerify`. No other new deps.
+
+## 3. New module: `src/api/oauth/`
+
+### `oauth-config.ts`
+
+```ts
+export interface OAuthConfig { issuer: string; jwksUrl: string; audience: string }
+export function oauthConfig(): OAuthConfig | null
+```
+
+Returns `null` unless `OAUTH_ENABLED === 'true'` and `OAUTH_ISSUER` is set. `jwksUrl` defaults to `${issuer}/oauth2/jwks`. `audience` defaults to `https://mcpscraper.dev/mcp`.
+
+### `verify-bearer.ts`
+
+```ts
+export interface OAuthIdentity { subject: string; email: string | null; scopes: string[] }
+export async function verifyOAuthBearer(token: string): Promise<OAuthIdentity | null>
+```
+
+Implementation requirements:
+- Module-level `createRemoteJWKSet(new URL(config.jwksUrl))` — jose caches keys internally; set `cooldownDuration: 30_000`, `cacheMaxAge: 3_600_000`.
+- `jwtVerify(token, jwks, { issuer: config.issuer, audience: config.audience, algorithms: ['RS256', 'ES256'], clockTolerance: 60 })` — pinning `algorithms` rejects `alg: none` and downgrade attacks by construction.
+- Extract `subject = payload.sub`, `email = payload.email ?? null` (WorkOS includes it), `scopes = (payload.scope ?? '').split(' ').filter(Boolean)`.
+- Any verification error → return `null` (caller converts to 401). Never throw to the route.
+
+### `resolve-user.ts`
+
+```ts
+export async function resolveOAuthUser(identity: OAuthIdentity): Promise<User | null>
+```
+
+Resolution order (each step returns on hit):
+1. `getUserByOauthSubject(identity.subject)` — fast path.
+2. If `identity.email`: `getUserByEmail(email)` → if found, `linkOauthSubject(user.id, identity.subject)` and return (links existing dashboard accounts to their Google identity by verified email — WorkOS only emits verified emails).
+3. Auto-provision: `createUser({ email, name: null })` (existing helper; generates `api_key`), then `grantSignupCredit(user.id)` (existing, idempotent), then `linkOauthSubject`. OAuth signups get the same 500-credit welcome grant as dashboard signups.
+4. No email claim and no subject match → return `null` (401; we never create anonymous accounts).
+
+## 4. DB changes — `src/api/db.ts`
+
+Migration (append to the existing `migrate()` DDL):
+
+```sql
+ALTER TABLE users ADD COLUMN oauth_subject TEXT;
+CREATE UNIQUE INDEX IF NOT EXISTS idx_users_oauth_subject ON users(oauth_subject) WHERE oauth_subject IS NOT NULL;
+```
+
+Guard the `ALTER` with the existing duplicate-column try/catch pattern used elsewhere in `migrate()`. New helpers, same row-mapping style as `getUserByApiKey`:
+
+```ts
+export async function getUserByOauthSubject(subject: string): Promise<User | undefined>
+export async function getUserByEmail(email: string): Promise<User | undefined>   // exists? reuse if so
+export async function linkOauthSubject(userId: number | bigint, subject: string): Promise<void>
+```
+
+Add `oauth_subject: string | null` to the `User` type and every row-mapper that builds `User` (`rowToUser`, the sweep mapper in `credit-operations.ts`).
+
+## 5. Protected Resource Metadata — new route
+
+New file `src/api/oauth/metadata-routes.ts`:
+
+```ts
+export const oauthMetadataApp = new Hono()
+oauthMetadataApp.get('/oauth-protected-resource', handler)
+oauthMetadataApp.get('/oauth-protected-resource/mcp', handler)   // path-scoped variant some clients request
+```
+
+Handler returns 200 JSON (404 when `oauthConfig()` is null):
+
+```json
+{
+  "resource": "https://mcpscraper.dev/mcp",
+  "authorization_servers": ["<OAUTH_ISSUER>"],
+  "bearer_methods_supported": ["header"],
+  "scopes_supported": ["mcp:tools"],
+  "resource_name": "MCP Scraper"
+}
+```
+
+Mount in `src/api/server.ts`: `app.route('/.well-known', oauthMetadataApp)`. Add to `vercel.json` rewrites: `{ "source": "/.well-known/:path*", "destination": "/api/index" }`.
+
+## 6. `src/mcp/mcp-routes.ts` changes
+
+### `mcpAuthError()` — upgrade the header
+
+```ts
+'WWW-Authenticate': `Bearer realm="mcp-scraper", resource_metadata="https://mcpscraper.dev/.well-known/oauth-protected-resource", error="invalid_token", error_description="Pass an MCP Scraper API key as x-api-key, or authenticate via OAuth"`
+```
+
+The `resource_metadata` parameter is what tells claude.ai where to start discovery (RFC 9728 §5.1). The live HTTP test already asserts `/Bearer/`; extend it to assert `/resource_metadata/`.
+
+### `requireMcpCallerKey()` → `requireMcpCaller()`
+
+Return type changes from `string | Response` to `{ user: User; callerKey: string } | Response`. Logic:
+
+```ts
+const xApiKey = header('x-api-key')
+const bearer  = Authorization Bearer value
+if (xApiKey) → getUserByApiKey(xApiKey) → { user, callerKey: xApiKey }
+else if (bearer && bearer.split('.').length === 3 && oauthConfig()) {
+  const identity = await verifyOAuthBearer(bearer)
+  if (!identity) return mcpAuthError()
+  const user = await resolveOAuthUser(identity)
+  if (!user) return mcpAuthError()
+  return { user, callerKey: user.api_key }          // executor bills/authorizes as this user
+}
+else if (bearer) → getUserByApiKey(bearer) → { user, callerKey: bearer }   // existing sk_ Bearer path
+else return mcpAuthError()
+```
+
+The three-dot heuristic routes JWTs to OAuth verification and `sk_*` keys to the existing lookup; order guarantees API keys never hit the JWT path. The handler body then uses `callerKey` exactly where it used the old return value — `new HttpMcpToolExecutor(baseUrl, callerKey)`. This is the no-passthrough rule made concrete: the OAuth token authenticates the session, but all internal REST work runs on the user's own API key.
+
+## 7. WorkOS dashboard configuration (ops, no code)
+
+1. Create WorkOS account → AuthKit → note the issuer `https://<subdomain>.authkit.app`.
+2. Enable **Dynamic Client Registration** (Applications → Configuration). Without this claude.ai cannot register; this is the most commonly missed step.
+3. Enable login methods: Google OAuth + email/password.
+4. Confirm access tokens are JWTs and the JWKS endpoint responds: `curl ${issuer}/oauth2/jwks`.
+5. Confirm AS metadata: `curl ${issuer}/.well-known/oauth-authorization-server` — must list `code` response type, PKCE `S256`, and the registration endpoint.
+6. Set the four env vars in Vercel + `.env`, redeploy.
+
+## 8. Tests
+
+### `tests/unit/oauth-verify-bearer.test.ts`
+
+Self-mint tokens with `jose.generateKeyPair('RS256')` + a local JWKS served via vitest mock of `createRemoteJWKSet` (inject key pair). Cases:
+- valid token → identity with subject/email/scopes
+- expired (`exp` in past, beyond 60s tolerance) → null
+- wrong `aud` → null
+- wrong `iss` → null
+- tampered signature → null
+- `alg: 'HS256'` (and `none`) → null
+- missing `sub` → null
+
+### `tests/unit/oauth-resolve-user.test.ts`
+
+Mock the db helpers. Cases: subject fast-path; email-link path (asserts `linkOauthSubject` called once); auto-provision path (asserts `createUser` + `grantSignupCredit` + `linkOauthSubject`); no-email no-match → null.
+
+### `tests/unit/oauth-metadata.test.ts`
+
+With env set: 200 + exact JSON shape, both paths. Without: 404.
+
+### Live (`tests/live/mcp/mcp-http-protocol.live.test.ts` additions)
+
+- 401 `WWW-Authenticate` includes `resource_metadata` (extend existing test).
+- `GET /.well-known/oauth-protected-resource` → 200 with `authorization_servers` non-empty (skip when `OAUTH_ISSUER` unset).
+- Full token flow live test: gated on `OAUTH_TEST_TOKEN` env (a token minted manually via WorkOS test client); asserts `tools/list` succeeds with only the Bearer JWT.
+
+### Manual E2E checklist (release gate)
+
+In claude.ai → Settings → Connectors → Add custom connector → `https://mcpscraper.dev/mcp` → expect AuthKit login screen → Google login → tools appear → run `credits_info` → verify the ledger shows the call against the auto-provisioned/linked account.
+
+## 9. Rollout phases
+
+1. **Code, dark** — §§1–6 + unit tests, `OAUTH_ENABLED` unset in prod. Everything is inert; ship in a normal release. (Includes the harmless `WWW-Authenticate` upgrade — only adds a parameter.)
+2. **WorkOS config + staging** — §7; flip `OAUTH_ENABLED=true` in prod (the gate makes this a config change, not a deploy); run live tests.
+3. **Manual E2E** — §8 checklist with claude.ai.
+4. **Docs surfaces** — README auth section; dashboard API-Key tab gets a "Connect via OAuth (claude.ai)" row in the client snippets (`public/app.jsx`, same component as the existing per-client snippets); `public/skills/mcp-scraper/skill.md` Authentication section gains one sentence.
+5. **Version bump + release** per release-workflow (vercel first, then npm — though npm package is unaffected; stdio never uses OAuth).
+
+## 10. Alternate AS: Clerk
+
+Identical resource-server code (that's the point of §3's isolation). Differences: issuer is `https://<app>.clerk.accounts.dev` (or custom domain), JWKS at `${issuer}/.well-known/jwks.json` (set `OAUTH_JWKS_URL` explicitly), enable "Dynamic client registration" in Clerk's OAuth applications settings, and confirm Clerk session tokens carry `aud` — if Clerk's token template needs a custom claim for audience, configure a JWT template named `mcp` with `aud: "https://mcpscraper.dev/mcp"`.
+
+## 11. Explicitly out of scope
+
+- Self-hosted authorization server (login UI, `/oauth2/token`, DCR storage, consent screens on Turso). Feasible later precisely because §3 isolates the AS contract; revisit only if vendor cost or control becomes a problem.
+- Scope-based tool authorization (per-tool scopes like `mcp:harvest`). Phase-2 candidate; `scopes` is already plumbed through `OAuthIdentity` so the data is there.
+- OAuth for the raw REST API (non-MCP). API keys remain the REST contract.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-scraper",
-  "version": "0.1.9",
+  "version": "0.2.0",
   "description": "MCP server for MCP Scraper web intelligence tools",
   "type": "module",
   "main": "./dist/index.cjs",
@@ -16,7 +16,8 @@
   "bin": {
     "paa-harvest": "dist/bin/paa-harvest.js",
     "paa-api": "dist/bin/api-server.js",
-    "mcp-scraper": "dist/bin/mcp-stdio-server.js"
+    "mcp-scraper": "dist/bin/mcp-stdio-server.js",
+    "browser-agent": "dist/bin/browser-agent-stdio-server.js"
   },
   "files": [
     "dist",