mcp-scraper 0.38.2 → 0.40.1

package/docs/specs/deferred-work-spec.md

@@ -1,86 +0,0 @@
-# Deferred Work Spec
-
-Items deliberately deferred from the June 2026 release train (0.1.7 → 0.1.9), specified to execution detail. Each section is independently shippable. Ordered by recommended priority.
-
-## 1. Ops: finish the vendor env var rename (no release needed)
-
-Code already reads `BROWSER_SERVICE_API_KEY` / `BROWSER_SERVICE_PROXY_ID` first and falls back to `KERNEL_API_KEY` / `KERNEL_PROXY_ID` (`src/lib/browser-service-env.ts`). Remaining steps are pure ops:
-
-1. Local: add to `.env`: `BROWSER_SERVICE_API_KEY=<same value as KERNEL_API_KEY>` and `BROWSER_SERVICE_PROXY_ID=<same as KERNEL_PROXY_ID>`.
-2. Vercel: `vercel env add BROWSER_SERVICE_API_KEY production` (paste value), same for `BROWSER_SERVICE_PROXY_ID`; redeploy (`vercel --prod --yes`).
-3. Verify: `curl -s https://mcpscraper.dev/me -H "x-api-key: $KEY"` still 200 and a `harvest_paa` smoke call succeeds.
-4. Only after a full green release cycle: remove `KERNEL_API_KEY`/`KERNEL_PROXY_ID` from Vercel and `.env`, then delete the fallback reads in `src/lib/browser-service-env.ts:2` and `:7` and the `?? process.env.KERNEL_API_KEY` in `src/api/env.ts:35`.
-5. Update live test requirements at the same time as step 4: `tests/live/mcp/mcp-stdio-protocol.live.test.ts:23` and `tests/live/mcp/mcp-http-protocol.live.test.ts:9` — change `requireLiveEnv([... 'KERNEL_API_KEY'])` to `'BROWSER_SERVICE_API_KEY'`.
-
-## 2. Ops: repair the git remote
-
-`git push` fails with "Repository not found"; release commits 0.1.6–0.1.9 exist only locally (single point of failure on this machine).
-
-1. Decide: fix the existing `origin` URL (`git remote -v`, then `git remote set-url origin <correct-url>`) or create a fresh private repo (`gh repo create vilovieta/mcp-scraper --private --source=. --push`).
-2. Push `main` and verify all four release commits land.
-3. Update the `release-workflow` memory note once pushes work.
-
-## 3. Hosted deep-harvest progress (deferred from task #7)
-
-**Problem:** `harvest_paa` calls budgeted up to 280 s return nothing until completion. The hosted transport runs stateless JSON mode (`enableJsonResponse: true`, `mcp-routes.ts`), which cannot stream `notifications/progress`.
-
-**Design — poll-based progress via the existing async REST API (works on BOTH transports):**
-
-1. `src/mcp/http-mcp-tool-executor.ts` — new private method `callWithPolling(input, onProgress)`:
-   - `POST /harvest` (existing async endpoint, returns `202 { job_id }`).
-   - Poll `GET /jobs/{job_id}` every 5 s up to the existing tiered budget from `harvestTimeoutBudget`.
-   - Each poll, derive progress from the job's `attempts` array (already returned): `onProgress({ progress: attempts.at(-1)?.question_count ?? 0, total: input.maxQuestions })`.
-   - On `status: 'done'` return the result in the existing `CallToolResult` shape; on `failed`/`cancelled` return the structured error (shape already produced by `harvestProblemResponse`).
-2. `src/mcp/paa-mcp-server.ts` — the `harvest_paa` handler gains the SDK's progress plumbing: the third callback argument `extra` exposes `sendNotification` and `_meta.progressToken`; when a token is present, forward `onProgress` events as `notifications/progress { progressToken, progress, total }`.
-3. stdio transport: works as-is (long-lived connection).
-4. Hosted transport: flip `mcp-routes.ts` to streaming mode — `enableJsonResponse: false` — so responses go out as SSE and interleaved progress notifications are legal. Keep `sessionIdGenerator: undefined` (still stateless per-request). Verify Vercel function streaming passes SSE through `api/index.ts`'s `streamResponse` (it already streams body chunks).
-5. Gate: only use polling mode when `maxQuestions > 50`; smaller calls keep the current single-shot `/harvest/sync` path.
-6. Tests: unit — fake fetch sequencing 202 → running → done, assert progress callbacks; live — stdio harvest with `maxQuestions: 60` asserting ≥1 progress notification received by the harness (add `onNotification` capture to `tests/live/harness/mcp-protocol-client.ts`).
-
-**Estimated blast radius:** executor, server builder, mcp-routes, harness, 2 test files. No REST API changes.
-
-## 4. Full OAuth 2.1 on hosted /mcp (deferred from task #9)
-
-**Superseded by `docs/specs/oauth-mcp-spec.md`** — decision locked (WorkOS AuthKit, Clerk as documented alternate, self-hosted out of scope), full implementation spec with files, functions, env vars, tests, and rollout phases. The decision matrix below is retained for historical context.
-
-Shipped already: `WWW-Authenticate: Bearer` on 401 (`mcp-routes.ts:mcpAuthError`). Remaining work requires a **product decision** before any code: which authorization server?
-
-| Option | Effort | Notes |
-|---|---|---|
-| WorkOS / Auth0 in front | Low | Fastest path to claude.ai connector compatibility; monthly cost |
-| Self-rolled AS on Turso (authorization codes + PKCE tables) | High | No vendor; significant security surface — not recommended |
-| Stay API-key only | Zero | Forecloses web-connector onboarding; revisit when demand exists |
-
-When a decision lands, the implementation is:
-
-1. `GET /.well-known/oauth-protected-resource` (RFC 9728) served from `server.ts`: `{ resource: "https://mcpscraper.dev/mcp", authorization_servers: ["<AS issuer>"], bearer_methods_supported: ["header"] }`. Add a `vercel.json` rewrite for the path.
-2. `src/mcp/mcp-routes.ts` `requireMcpCallerKey`: accept a Bearer JWT in addition to API keys — validate signature against the AS JWKS (cache keys 1 h), require `aud` containing `https://mcpscraper.dev/mcp` (RFC 8707 resource binding), map the token's subject to a user row via a new `users.oauth_subject` column (migration in `db.ts`).
-3. Keep `x-api-key` working forever (NPX clients, REST customers).
-4. Tests: unit for token validation paths (expired/wrong-aud/garbage); live test gated on AS test-client credentials in env.
-
-## 5. outputSchema for the capture tools (deliberately excluded)
-
-`capture_serp_snapshot` / `capture_serp_page_snapshots` return raw passthrough of the SERP Intelligence product payload, which is still evolving with Phoenix. Adding `outputSchema` now would either freeze that payload or break on every product iteration. Revisit when the Phoenix capture contract stabilizes; at that point: schema in `mcp-tool-schemas.ts`, a formatter (currently none — executor result passes straight through), and registration updates in `mcp-routes.ts:registerSerpIntelligenceCaptureTools`.
-
-## 6. Maintenance follow-ups (small, batch into any release)
-
-1. `public/skill.md` / `public/codex-skill.md` (REST-mode skill files): regenerate the `maxQuestions` guidance to say cap 200 (currently 1–150 in the parsing table) — `public/skill.md` PAA Harvest Mode STEP 1 table row `maxQuestions`.
-2. `src/cli.ts:21` still names the flag `--kernel-api-key` (help text already neutral). Renaming the flag breaks invocations; add an alias `--browser-api-key` and hide the old one when commander supports it.
-3. `tests/live/harness/mcp-protocol-client.ts` `close()` uses `setTimeout(resolve, 3_000)` without unref — harmless in vitest, but unref it if the harness ever runs outside vitest.
-4. Internal naming (`BrowserDriver` kernel fields, DB columns `kernel_session_id`, `src/api/kernel-fetch.ts`, `src/kernel-proxy-resolver.ts`) intentionally unchanged — outbound sanitizers cover the boundary. Do NOT rename DB columns without a migration plan; the sanitizers make it unnecessary.
-
-## 7. API Forge
-
-Specified separately in `docs/specs/api-forge-spec.md`. Recommended to schedule after §1 and §2 (ops hygiene first), before §3/§4 (it produces revenue surface; progress/OAuth polish can follow demand).
-
-## 8. SEO CLI growth workflows
-
-Specified separately in `docs/specs/seo-cli-growth-roadmap-spec.md`, with buildable sub-specs for agent wiring, workflow reports, agent-ready SEO packets, local competitive audits, and scheduled workflow APIs:
-
-- `docs/specs/cli-agent-wiring-spec.md`
-- `docs/specs/seo-workflow-runner-and-reports-spec.md`
-- `docs/specs/agent-ready-seo-packet-spec.md`
-- `docs/specs/local-competitive-audit-spec.md`
-- `docs/specs/scheduled-workflows-api-spec.md`
-
-Recommended priority: ship agent wiring and local workflow reports first, then the agent packet and local competitive audit workflows, then hosted schedules/API once report artifacts are durable.