mcp-scraper 0.38.2 → 0.40.1

package/docs/specs/api-forge-spec.md

@@ -1,234 +0,0 @@
-# API Forge — Implementation Spec
-
-Research-grounded API design intelligence inside MCP Scraper. A database-design tool asks about entities and relationships, then emits DDL. API Forge asks about resources, operations, and consumers — then emits OpenAPI, Zod schemas, SQL DDL, and **MCP tool definitions that comply with `docs/mcp-tool-quality-spec.md`** — with every design decision groundable in live web evidence harvested by the existing scraper tools.
-
-## The three-party design
-
-This is the core architecture decision and the creative center of the feature:
-
-| Party | Owns | Never does |
-|---|---|---|
-| **MCP Scraper server** | Methodology (phase machine, question cards), session state, research execution, artifact generation, critique rules | Conversation with the human |
-| **Calling LLM** (Claude, etc.) | Interviewing the human, translating answers into structured form, narrating findings | Design methodology, state |
-| **Scraper tools** (existing) | Evidence: PAA questions developers ask, competitor API doc conventions, domain entity discovery | — |
-
-The server returns *question cards* and *synthesis instructions*; the calling model conducts the interview and reports structured answers back. This works on every MCP host (no dependency on elicitation support) and keeps the methodology versioned server-side.
-
-## Tool surface (5 new MCP tools)
-
-All registered in `src/mcp/paa-mcp-server.ts`, schemas in `src/mcp/mcp-tool-schemas.ts`, formatters in `src/mcp/mcp-response-formatter.ts`. All have `outputSchema` + `structuredContent` (these are chaining tools by definition). Annotations: `api_critique` is `readOnlyHint: true`; the four session tools are `readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true` (they mutate server-side session state).
-
-### 1. `api_design_start`
-
-```
-inputSchema: {
-  name:    z.string().min(1).describe('Working name of the API, e.g. "Acme Bookings API"'),
-  purpose: z.string().min(1).describe('One-sentence purpose in the user\'s words'),
-  domain:  z.string().min(1).describe('Business domain, e.g. "salon appointment booking" — used to seed research'),
-  research: z.boolean().default(true).describe('Run an automatic domain research pass (PAA + SERP) before the first interview card. Costs credits per the cost table.'),
-}
-```
-
-Behavior: creates a session, optionally runs Phase-0 research (internal calls to `harvest()` with `maxQuestions: 15` and query `"${domain} api"` — direct function call, not HTTP; billed via existing `paa` ledger op), and returns the first question card.
-
-Description (model-facing, per quality spec): "Start an API design session. Returns a sessionId, optional domain research findings, and the first interview card — a batch of design questions. Interview the user conversationally using the card, then submit their answers with api_design_answer. Do not invent answers the user did not give."
-
-`outputSchema`: `{ sessionId, phase, card: CardOutput, research: { paaQuestions: string[], topCompetitorUrls: string[] } | null }`
-
-### 2. `api_design_answer`
-
-```
-inputSchema: {
-  sessionId: z.string().min(1),
-  answers:   z.array(z.object({
-    questionId: z.string().min(1),
-    value:      z.string().min(1).describe('The user\'s answer, verbatim or faithfully summarized'),
-    skipped:    z.boolean().default(false),
-  })).min(1),
-}
-```
-
-Behavior: validates answers against the current card, folds them into the `DesignModel` via per-question reducers, advances the phase machine when the card is satisfied, returns the next card (or `phase: 'complete'`). Free (no debit) — answers are state writes only.
-
-`outputSchema`: `{ sessionId, phase, accepted: number, designSummary: DesignSummaryOutput, card: CardOutput | null, complete: boolean }`
-
-### 3. `api_design_research`
-
-```
-inputSchema: {
-  sessionId:      z.string().min(1),
-  competitorUrls: z.array(z.string().url()).max(5).optional().describe('API documentation URLs to mine for conventions (pagination idiom, error shape, naming, auth)'),
-  topic:          z.string().optional().describe('Free-form research topic, e.g. "webhook retry best practices" — runs a PAA harvest'),
-}
-```
-
-Behavior: for each competitor URL, runs the existing `extract_url` pipeline internally, then a **deterministic convention extractor** (no server-side LLM): regex/heading analysis of the extracted Markdown detecting pagination style (`cursor|page|offset` token frequency), error envelope (`"error"`/`"errors"`/problem+json mentions), naming case (snake vs camel ratio in code blocks), auth scheme mentions. For `topic`, runs a PAA harvest. Results are stored as `evidence[]` entries on the session and returned with a `synthesisInstruction` string telling the calling model how to present findings and which open design questions the evidence bears on. Billed: `page_scrape` per URL, `paa` per question — existing ledger ops, no new rates needed.
-
-### 4. `api_design_blueprint`
-
-```
-inputSchema: {
-  sessionId: z.string().min(1),
-  formats:   z.array(z.enum(['openapi', 'zod', 'sql', 'mcp_tools', 'readme'])).min(1)
-               .default(['openapi', 'zod', 'sql', 'mcp_tools', 'readme']),
-}
-```
-
-Behavior: runs the generators (below) against the materialized `DesignModel`. Allowed mid-interview (partial blueprint from whatever is designed so far — this enables the "show me where we are" loop). Returns artifacts inline as fenced code blocks in `content` plus `structuredContent.artifacts: [{ format, filename, source }]`. On stdio, each artifact is also saved through the existing `saveFullReport` path, which makes blueprints appear in the `report://` resources list for free. New rate: `api_blueprint` 1 000 mc (1 credit) per generation.
-
-### 5. `api_critique`
-
-```
-inputSchema: {
-  openapiUrl: z.string().url().optional().describe('URL of an existing OpenAPI/Swagger document, or API docs page'),
-  openapiSource: z.string().optional().describe('Pasted OpenAPI YAML/JSON when no URL is available'),
-  audience:   z.enum(['human', 'service', 'ai_agent']).default('ai_agent').describe('Score the API for this consumer. ai_agent applies MCP-readiness rules.'),
-}
-```
-
-Behavior: fetch (via internal `extract_url` when URL) → parse OpenAPI → run the critique rule table → return scored findings `[{ ruleId, severity: 'P0'|'P1'|'P2', location, finding, fix }]`. The `ai_agent` audience adds the MCP-readiness rules derived from `docs/mcp-tool-quality-spec.md` (description quality, enum documentation, error actionability, schema-encoded defaults/caps). This is the standalone marketing wedge: "run our quality spec against *your* API." New rate: `api_critique` 2 000 mc.
-
-## Reverse Forge (the second creative wedge — Phase 6, optional)
-
-`api_design_start` accepts an optional `importUrl`: point it at an existing API's docs site; the server runs `map_site_urls` + `extract_url` over the docs internally, reconstructs a partial `DesignModel` from the conventions extractor, and starts the interview at the *gaps* instead of from zero. The session then supports "extend this API with X" with consistency checks against the inferred conventions (e.g. "your existing endpoints use cursor pagination and snake_case; the new webhooks resource will follow"). Implemented as `meta.importedFrom` on the session plus a `seedDesignFromDocs(extracts: ExtractResult[]): Partial<DesignModel>` function in the conventions extractor module.
-
-## The interview engine
-
-### `src/forge/design-model.ts` — exact types
-
-```ts
-export interface DesignField { name: string; type: 'string'|'number'|'boolean'|'timestamp'|'json'|'ref'; refEntity?: string; required: boolean; description: string }
-export interface DesignEntity { name: string; plural: string; identity: 'uuid'|'slug'|'int'; fields: DesignField[] }
-export interface DesignRelationship { from: string; to: string; kind: 'one_to_one'|'one_to_many'|'many_to_many'; ownership: 'embedded'|'referenced' }
-export interface DesignOperation { entity: string; verb: 'list'|'get'|'create'|'update'|'delete'|'action'; actionName?: string; idempotent: boolean; paginated: boolean; description: string }
-export interface DesignContracts { errorShape: 'problem_json'|'envelope'|'custom'; pagination: 'cursor'|'offset'|'none'; naming: 'snake'|'camel'; envelope: boolean }
-export interface DesignAuth { scheme: 'api_key'|'oauth2'|'jwt'|'none'; scopes: string[] }
-export interface DesignEvidence { source: 'paa'|'serp'|'competitor_doc'|'user'; url: string | null; finding: string; appliedTo: string }
-export interface DesignModel {
-  meta: { name: string; purpose: string; domain: string; consumers: Array<'human'|'service'|'ai_agent'>; importedFrom?: string }
-  entities: DesignEntity[]
-  relationships: DesignRelationship[]
-  operations: DesignOperation[]
-  contracts: DesignContracts
-  auth: DesignAuth
-  evidence: DesignEvidence[]
-}
-```
-
-### `src/forge/question-cards.ts` — phase machine as data
-
-```ts
-export type ForgePhase = 'consumers'|'entities'|'relationships'|'operations'|'contracts'|'auth'|'review'|'complete'
-export interface QuestionCard {
-  phase: ForgePhase
-  intro: string                     // one paragraph the model relays before asking
-  questions: Array<{
-    id: string                      // e.g. 'entities.list'
-    ask: string                     // the question, written to be read aloud
-    why: string                     // rationale the model can offer if asked
-    expects: 'free_text'|'entity_list'|'choice'|'yes_no'
-    choices?: string[]
-    skippable: boolean
-    appliesIf?: (model: DesignModel) => boolean   // skip logic, e.g. m2m junction question only if any many_to_many
-  }>
-}
-export const CARDS: Record<ForgePhase, QuestionCard>
-```
-
-Phase order and headline questions (full card text lives in the file):
-
-1. **consumers** — "Who calls this API: humans via a UI, backend services, or AI agents?" (multi-choice; `ai_agent` switches on MCP generation + agent-readability rules). "What must never be exposed?" (seeds redaction rules — mirrors this repo's vendor-concealment pattern).
-2. **entities** — "Name the things your API manages, singular nouns." Then per entity: identity strategy, 3–7 core fields. Reducer: `entities.list` answer is parsed as comma/newline-separated nouns; per-entity follow-up cards are generated dynamically (`entities.fields.{name}`).
-3. **relationships** — for each entity pair that co-occurred in answers: "Does an X own many Y?" Cardinality + embedded-vs-referenced.
-4. **operations** — per entity: which of list/get/create/update/delete, plus domain actions ("cancel a booking" → `POST /bookings/{id}/cancel`, `idempotent: false`).
-5. **contracts** — pagination, error shape, naming. **This is where research lands**: if the session has competitor evidence, the card's `intro` includes "Competitors X and Y both use cursor pagination" and the question defaults accordingly.
-6. **auth** — scheme + scopes; if consumers includes `ai_agent`, ask about per-call cost/metering (seeds a credits-style design like this repo's).
-7. **review** — the server returns a `DesignSummary`; the model walks the user through it; any "change X" answers route back to the owning phase.
-
-### `src/forge/interview-engine.ts`
-
-```ts
-export function nextCard(model: DesignModel, phase: ForgePhase): { card: QuestionCard | null; phase: ForgePhase }
-export function applyAnswers(model: DesignModel, phase: ForgePhase, answers: Answer[]): { model: DesignModel; errors: string[] }
-export function designSummary(model: DesignModel): DesignSummaryOutput
-```
-
-Pure functions — no I/O — so the whole engine is unit-testable with golden sessions.
-
-## Generators — `src/forge/generators/`
-
-Each is `(model: DesignModel) => { filename: string; source: string }`, pure, individually golden-tested.
-
-- `openapi.ts` — OpenAPI 3.1 YAML. Paths from operations (`GET /bookings`, `POST /bookings/{id}/cancel`), components.schemas from entities, error responses per `contracts.errorShape`, pagination params per `contracts.pagination`, securitySchemes per auth.
-- `zod.ts` — one Zod schema per entity + per-operation input schemas, naming per `contracts.naming`.
-- `sql.ts` — SQLite-flavored DDL: one table per entity, junction tables for `many_to_many`, FK columns for `referenced` one-to-many, `TEXT` ISO timestamps (house style of this repo's `db.ts`).
-- `mcp-tools.ts` — the dogfood crown: emits a `registerTool` block per operation following `docs/mcp-tool-quality-spec.md` — model-instructing description ("Use this when…"), schema-encoded defaults/caps, annotations (`readOnlyHint: true` for list/get), `outputSchema` for chaining ops, error-shape guidance. Generated against the same `liveWebToolAnnotations` helper pattern used in `paa-mcp-server.ts`.
-- `readme.ts` — install + auth + per-endpoint docs with curl examples.
-
-## Persistence — `src/api/db.ts` additions
-
-```sql
-CREATE TABLE IF NOT EXISTS api_forge_sessions (
-  id          TEXT PRIMARY KEY,
-  user_id     INTEGER NOT NULL,
-  name        TEXT NOT NULL,
-  phase       TEXT NOT NULL DEFAULT 'consumers',
-  status      TEXT NOT NULL DEFAULT 'active',
-  design_json TEXT NOT NULL DEFAULT '{}',
-  created_at  TEXT NOT NULL,
-  updated_at  TEXT NOT NULL
-);
-CREATE TABLE IF NOT EXISTS api_forge_events (
-  id           INTEGER PRIMARY KEY AUTOINCREMENT,
-  session_id   TEXT NOT NULL,
-  kind         TEXT NOT NULL,        -- 'answers' | 'research' | 'blueprint'
-  payload_json TEXT NOT NULL,
-  created_at   TEXT NOT NULL
-);
-```
-
-`design_json` is the materialized DesignModel; events are the audit trail (enables replay/undo later). DB helpers: `createForgeSession`, `getForgeSession`, `updateForgeSession`, `appendForgeEvent` — same row-mapping style as `getJob`.
-
-## Routes — `src/api/api-forge-routes.ts`
-
-`forgeApp = new Hono<ApiKeyEnv>()` mounted at `/api-forge` in `server.ts` and added to `vercel.json` rewrites (`{ "source": "/api-forge/:path*", "destination": "/api/index" }`). Five POST routes mirroring the tools; `createApiKeyAuth`; debit-before-work + refund-on-failure exactly like `maps-routes.ts`; errors in the structured `{error, error_code, retryable}` shape. Session ownership check on every route (`session.user_id === user.id` else 404).
-
-## Rates — `src/api/rates.ts` additions
-
-```ts
-api_forge_start: 2_000,   // 2 credits, includes Phase-0 research overhead
-api_blueprint:   1_000,
-api_critique:    2_000,
-```
-
-Plus `CREDIT_COST_CATALOG` entries with aliases (`'api design'`, `'blueprint'`, `'critique'`), `LedgerOperation.API_FORGE_START / API_BLUEPRINT / API_CRITIQUE` + paired `_REFUND` ops (the `ledger-refund-keys` test enforces pairing).
-
-## MCP wiring
-
-- Executor: 5 methods on `IMcpToolExecutor`… **no** — these are a separate concern; add `IApiForgeToolExecutor` interface in `IMcpToolExecutor.ts` (same pattern as `ISerpIntelligenceToolExecutor`), implemented by `HttpMcpToolExecutor` (`this.call('/api-forge/start', input)` etc.).
-- Registration: `registerApiForgeTools(server, executor)` in `paa-mcp-server.ts`, called from both transports (these tools work identically hosted and stdio — session state is server-side).
-- Formatters: each returns a Markdown report (`oneBlock`) + `structuredContent`. The blueprint formatter renders each artifact as a fenced block and saves per-artifact files on stdio.
-
-## Tests
-
-- `tests/unit/forge-interview-engine.test.ts` — phase transitions, skip logic, reducer correctness, a full golden session (fixture answers → expected DesignModel).
-- `tests/unit/forge-generators.test.ts` — golden outputs per generator from a fixture DesignModel; the `mcp-tools` generator output is itself grep-asserted for quality-spec markers (`readOnlyHint`, `.default(`, `.max(`).
-- `tests/unit/forge-conventions-extractor.test.ts` — pagination/error/naming detection from fixture doc Markdown.
-- `tests/unit/mcp-server-registration.test.ts` — tool count 13 → 18 (stdio) / 20 (hosted); annotations table updated.
-- `tests/live/mcp/api-forge.live.test.ts` — start → answer × 2 → blueprint over stdio against the test server; asserts artifacts parse (YAML loads, DDL contains `CREATE TABLE`).
-
-## Build phases
-
-1. **Engine** — design-model.ts, question-cards.ts, interview-engine.ts + unit tests. No I/O. (Largest thinking, smallest risk.)
-2. **Persistence + routes** — db.ts tables, api-forge-routes.ts (start/answer only), rates, vercel.json rewrite.
-3. **Generators** — all five + golden tests; blueprint route + tool.
-4. **Research** — conventions extractor, internal harvest/extract calls, evidence folding, research tool.
-5. **Critique** — rule table + tool (independent of sessions; can ship before 4 if desired).
-6. **Reverse Forge** — importUrl seeding (optional, after 1–5 prove out).
-7. **Release** — quality-spec Definition-of-Done sweep: README, `public/skills/mcp-scraper/skill.md` tool list, registration tests, live tests, version bump, vercel deploy then npm publish.
-
-## Open decisions (resolve before Phase 2)
-
-1. Session TTL/limits: cap active sessions per user (proposal: 10) and expire after 30 days (cron at `/cron/tick` already exists).
-2. Should `api_design_answer` bill micro-credits to deter abuse, or stay free? (Proposal: free; starts are the gate.)
-3. Server-side LLM synthesis: deliberately excluded — the calling model synthesizes, the server stays deterministic. Revisit only if convention extraction proves too weak on real competitor docs.