mcp-scraper 0.3.14 → 0.3.16

package/dist/{worker-FG7ZWEGA.js → worker-2WVKKCC7.js}

@@ -4,7 +4,7 @@ import {
   createHarvestAttemptRecorder,
   harvestProblemResponse,
   serializeHarvestProblem
-} from "./chunk-AZ5PKU4F.js";
+} from "./chunk-ZAUMSBV3.js";
 import {
   claimPendingJob,
   completeJob,
@@ -12,7 +12,7 @@ import {
   debitMc,
   failJob,
   listHarvestAttempts
-} from "./chunk-DBQDG7EH.js";
+} from "./chunk-5H22TOXQ.js";
 import {
   harvest
 } from "./chunk-4OPKIDON.js";
@@ -125,4 +125,4 @@ export {
   startWorker,
   tickOnce
 };
-//# sourceMappingURL=worker-FG7ZWEGA.js.map
+//# sourceMappingURL=worker-2WVKKCC7.js.map

package/docs/final-tooling-spec.md

@@ -0,0 +1,206 @@
+# Final Tooling Spec — rename, server instructions, description optimization, catalog
+
+Execution blueprint consolidating the decisions reached: (A) server namespace rename
+`mcp-scraper → scraper`, (B) `*_workflow` rename of workflow-class tools, (C) a front-loaded
+**server-instructions block** carrying the cross-tool routing map, (D) a per-tool description
+optimization pass that moves routing *out* of descriptions, and (E) catalog integration.
+
+Grounded in: the SDK's `instructions?: string` option
+(`node_modules/@modelcontextprotocol/sdk/dist/esm/server/index.d.ts:15`), verified tool registrations
+in `src/mcp/paa-mcp-server.ts` and `src/mcp/browser-agent-mcp-server.ts`, and the executor map in
+`src/mcp/http-mcp-tool-executor.ts`.
+
+Governing principle (from `docs/mcp-tool-design-guide.md`): **the model routes on names + the
+server-instructions block before it fetches any schema.** Therefore the cross-tool routing map must
+live in the instructions block, never solely in per-tool descriptions that may never be fetched.
+
+---
+
+## Part A — Server namespace rename: `mcp-scraper` → `scraper`
+
+The client prepends `mcp__`, so today every tool is `mcp__mcp-scraper__…` (stutter). Renaming the
+**server name** to `scraper` yields `mcp__scraper__…`.
+
+### A.1 Exact edits — CORRECTED after verification
+
+> **The namespace `mcp__<key>__` comes from the CLIENT REGISTRATION KEY, not the McpServer `name`.**
+> Verified: `src/cli/agent-config.ts` registers the server with the key `mcp-scraper` via
+> `claude mcp add mcp-scraper …` and the Desktop config block `'mcp-scraper': {…}`. THAT key is what
+> Claude turns into `mcp__mcp-scraper__*`. Changing only `new McpServer({name})` does NOT change the
+> prefix. The full set of edits to actually rename the namespace:
+
+- `src/cli/agent-config.ts:54` — `['mcp', 'remove', 'mcp-scraper', …]` → `'scraper'`.
+- `src/cli/agent-config.ts:58` — `['mcp', 'add', 'mcp-scraper', …]` → `'scraper'`.
+- `src/cli/agent-config.ts:75` — Desktop config key `'mcp-scraper': {…}` → `'scraper': {…}`.
+- `scripts/build-mcpb.mjs:61` — `.mcpb` manifest server `name: 'mcp-scraper'` → `'scraper'` (Desktop
+  extension registration key). Line 50 (`mcp-scraper-desktop-extension`) is the extension *package* id; leave it.
+- `src/mcp/paa-mcp-server.ts:152` and `bin/mcp-scraper-combined-stdio-server.ts:61` — the McpServer
+  `name` is cosmetic self-description; update to `'scraper'` for consistency but it is NOT what drives the prefix.
+- **Tests:** `tests/unit/cli-agent-config.test.ts:32` asserts `'mcp-scraper'` — update.
+- **Migration is harder than a code change:** existing users already have the server registered under
+  `mcp-scraper` in *their* client config. Renaming the key here only affects NEW installs; existing
+  installs keep `mcp__mcp-scraper__*` until they `claude mcp remove mcp-scraper && claude mcp add scraper`
+  (or reinstall the `.mcpb`). Document this explicitly; consider shipping a migration note/command.
+
+### A.2 Leave alone (URL / artifact stability)
+- Bundle filenames/paths in `build-mcpb.mjs` (`mcp-scraper.mcpb`, `mcp-scraper-${version}.mcpb`,
+  `build/mcpb/mcp-scraper`, `dist/bin/mcp-scraper-combined-stdio-server.js`) — **keep**. These are
+  download URLs and build paths; changing them breaks existing download links for no namespace benefit.
+- Repo name, GitHub URLs — keep.
+
+### A.3 Breaking-change handling (REQUIRED — do not skip)
+Renaming the server changes **every tool identifier** clients see → invalidates:
+- User permission allowlists referencing `mcp__mcp-scraper__*`.
+- Any saved agent configs / scripted tool calls by full name.
+
+Mitigations:
+- **Do it in a single release, called out in the changelog** as a breaking namespace change with a
+  one-line migration ("re-allow tools; the namespace is now `mcp__scraper__…`").
+- Bump a **minor/major** version, not a patch.
+- Ship in the same release as Parts B–D so users re-onboard once, not repeatedly.
+
+---
+
+## Part B — Workflow-class tool renames (`*_workflow` suffix)
+
+Rule: a tool whose mental model is "run a multi-step workflow" carries `_workflow` so the keyword is in
+the name (matches a user saying "run the … workflow / research / directory"). Engine plumbing
+(`workflow_run/step/status/list/suggest/artifact_read`) is **not** renamed — it's infrastructure, not a
+user-named capability.
+
+### B.1 Rename map
+| Current name | New name | File(s) |
+|---|---|---|
+| `rank_tracker_blueprint` | `rank_tracker_workflow` | `paa-mcp-server.ts:346`, schema, manifest, catalog |
+| `browser_capture_fanout` | `query_fanout_workflow` | `browser-agent-mcp-server.ts:791`, `browser-agent-tool-schemas.ts:189`, manifest, catalog |
+| `directory_workflow` | *(unchanged — already correct)* | — |
+| *(new, optional)* | `deep_research_workflow` | thin wrapper → `workflow_run` for the `deep-research` workflow; register in `paa-mcp-server.ts` |
+
+### B.2 Per-tool change checklist (apply to each renamed tool)
+1. **Registration** — `server.registerTool('<new_name>', { … })` (the string literal).
+2. **Input schema** — rename the exported `…InputSchema` only if its name encodes the old tool name;
+   the schema *shape* is unchanged.
+3. **Manifest** — `src/mcp/mcp-tool-manifest.ts` `entry({ name: '<new_name>', … })`.
+4. **Executor** — `src/mcp/http-mcp-tool-executor.ts` method name only if it mirrors the tool name; the
+   **REST route string is unchanged** (rename is MCP-surface only, not the backend).
+5. **Formatter** — `format<Tool>` function name optional; wiring in `registerTool` handler updated.
+6. **Catalog** — update `mcp_name` in `docs/tool-catalog-spec.md` §3.2/§9 and the seed.
+7. **Lint/manifest generators** — re-run `npm run mcp:manifest` and `npm run mcp:lint-tools` to
+   regenerate `docs/mcp-tool-manifest.generated.*`.
+
+### B.3 Optional backward-compat (recommended for 1–2 releases)
+Register the **old name as a hidden alias** pointing at the same handler, so in-flight callers don't
+hard-break: `server.registerTool('<old_name>', { ...sameConfig, annotations: { deprecated: true } }, handler)`.
+Remove after one release. If you'd rather hard-cut, fold it into the Part A breaking release.
+
+---
+
+## Part C — Server-instructions block (the pre-fetch routing map)
+
+This is the highest-leverage change: a short string loaded **at session start** (alongside tool names),
+so the model routes correctly *before* fetching any schema.
+
+### C.1 New file — `src/mcp/server-instructions.ts`
+```ts
+export const SERVER_INSTRUCTIONS = `
+This server scrapes and analyzes web, social, search, maps, and site data. Pick a tool by NAME using
+this routing map; load its schema before calling.
+
+ROUTING
+- One web page → extract_url. Whole site (crawl + SEO) → extract_site. Just the URL list → map_site_urls.
+- Web search (SERP) → search_serp. Full SERP + People-Also-Ask detail → harvest_paa.
+- Google Maps: find places → maps_search; one place deep-dive + reviews → maps_place_intel.
+- YouTube: find/list videos → youtube_harvest; transcribe one → youtube_transcribe.
+- Facebook: find ads → facebook_ad_search; transcribe an ad → facebook_ad_transcribe; transcribe a
+  video → facebook_video_transcribe; page profile → facebook_page_intel.
+- Instagram: profile inventory → instagram_profile_content; one post/reel → instagram_media_download.
+- Workflows (multi-step): local directory → directory_workflow; rank blueprint → rank_tracker_workflow;
+  AI-answer citation fan-out (AEO) → query_fanout_workflow; deep research → deep_research_workflow.
+- Anything without a dedicated tool (e.g. Reddit, arbitrary logged-in sites) → the browser_* agent
+  (browser_open then navigate/read), and capture profiles via browser_profile_*.
+
+NOTES
+- Bulk/full-site crawls: call extract_site with rotateProxies:true for blocked/rate-limited sites; it
+  discovers URLs and returns a saved folder/artifact, not the full content inline.
+- Tools that open a browser session or transcribe media cost credits and take longer — prefer the
+  cheapest tool that answers the question (plain search/extract before browser agents).
+- Results may save large output to disk/artifact and return a summary + path; read the path for detail.
+`.trim()
+```
+> Keep it **cross-tool only** — no per-tool parameter detail (that's the fetched schema's job). Use
+> neutral capability language; never name the underlying browser/proxy vendor (concealment policy).
+
+### C.2 Wiring (both server construction sites)
+- `src/mcp/paa-mcp-server.ts:152`:
+  `new McpServer({ name: 'scraper', version: PACKAGE_VERSION }, { instructions: SERVER_INSTRUCTIONS })`
+- `bin/mcp-scraper-combined-stdio-server.ts:61`: same second argument.
+- Import `SERVER_INSTRUCTIONS` from `../src/mcp/server-instructions.js` (and `./server-instructions.js`).
+
+### C.3 The insight, enforced
+Per-tool descriptions must **NOT** be the only home for "which tool for which job." That guidance moves
+to `SERVER_INSTRUCTIONS` (always loaded). Per-tool descriptions keep only a **single** negative-space
+line for post-fetch tie-breaking (see Part D). This prevents the failure where the routing hint lives in
+a description the model never fetched.
+
+---
+
+## Part D — Tool-description optimization pass
+
+For every tool in `mcp-tool-schemas.ts` / `paa-mcp-server.ts` `registerTool` descriptions, rewrite to
+this shape (apply the `docs/mcp-tool-design-guide.md` checklist):
+
+1. **Line 1 = when-to-use**, in the user's words. ("Transcribe one YouTube video to text…")
+2. **Line 2 = side-effects/cost** if any. ("Opens a browser session; costs credits per minute.")
+3. **Params**: each param's `.describe()` states intent + bounds + default, mirroring the Zod schema.
+4. **One** negative-space line max. ("For a whole site use extract_site.") — the *broad* map lives in
+   §C, so do not enumerate the full tool family here.
+5. **Delete** redundant cross-tool prose and marketing. Shorter descriptions load faster and rank cleaner.
+
+Concrete trims to make now:
+- `extract_url` / `extract_site` / `map_site_urls`: each currently restates the others; cut to one
+  negative-space line each (the trio map is in §C).
+- `search_serp` vs `harvest_paa`: both hit `/harvest/sync`; clarify in one line each that
+  search_serp = organic results only, harvest_paa = + PAA/AI-Overview/SERP features.
+- Renamed workflow tools: rewrite line 1 to lead with the keyword now in the name.
+
+Re-run `npm run mcp:lint-tools` (the craft linter) after edits; fix any flagged tools.
+
+---
+
+## Part E — Catalog integration
+
+Reconcile `docs/tool-catalog-spec.md` with the renames, and ship the **static v1** (no DB) first:
+
+- Update catalog `mcp_name` values: `rank_tracker_blueprint→rank_tracker_workflow`,
+  `browser_capture_fanout→query_fanout_workflow`; add `deep_research_workflow`.
+- Categories stay **platform-first** (YouTube / Facebook / Instagram / Google Search / Google Maps /
+  Websites / Browser / Workflows), with **Reddit** served by the Browser category (no dedicated tool) —
+  or add a `reddit_*` named wrapper later if AI routing for Reddit is wanted (per the design guide).
+- **v1 implementation = static**: `src/api/catalog-seed.ts` exports `CATALOG`; `GET /catalog` in
+  `server.ts` returns it. Defer the 6-table DB schema (§2 of the catalog spec) to v2 (runtime editing).
+- Catalog params must mirror the live Zod schemas (field names/bounds) so the frontend posts valid bodies.
+
+---
+
+## Part F — Build order
+1. **Part C** (server instructions) — highest leverage, lowest risk, non-breaking. Ship first.
+2. **Part D** (description optimization) — non-breaking; pairs with C (move routing out → into C).
+3. **Part B** (workflow renames) — with hidden aliases (B.3) so it's non-breaking.
+4. **Part A** (server rename) — the one breaking change; batch into a single versioned release with a
+   changelog migration note. Run after B/C/D so users re-onboard once.
+5. **Part E** static catalog — independent; any time.
+6. Regenerate `docs/mcp-tool-manifest.generated.*`; run `npm run mcp:lint-tools`; full test + live tests;
+   then the release flow (3 version files, Vercel prod first, npm publish, rebuild `.mcpb`).
+
+## Part G — Verification checklist
+- [ ] `grep -rn "mcp-scraper'" src bin scripts` → only intentional (bundle filenames), server `name` = `scraper`.
+- [ ] No tool identifier appears in two namespaces; `npm run mcp:manifest` regenerated.
+- [ ] `SERVER_INSTRUCTIONS` loads on `initialize` (verify via an MCP inspector / the live protocol test).
+- [ ] Every renamed tool: registration + manifest + catalog `mcp_name` updated; old name aliased or cut.
+- [ ] No per-tool description contains the full cross-tool map (only one negative-space line).
+- [ ] No vendor name (browser/proxy provider) in instructions or any description.
+- [ ] `npm run mcp:lint-tools` clean; `vitest run tests/unit tests/contract` green.
+
+## Non-goals (this spec)
+DB-backed catalog (v2), the `reddit_*` / `deep_research_workflow` *implementations* beyond thin
+registration, per-plan tool gating, and the frontend rendering itself.

package/docs/mcp-tool-design-guide.md

@@ -0,0 +1,225 @@
+# Designing MCP Tools an AI Can Actually Find and Use
+
+A practical guide to building MCP servers whose tools an LLM (Claude, but the mechanics generalize)
+will **discover, select correctly, and invoke well**. It is written from how the model *actually*
+perceives tools at runtime — not how a human reads your docs page.
+
+The single thesis: **the model routes on what it can see, and it can see almost nothing until it asks.**
+Everything below follows from that.
+
+---
+
+## 1. How the model actually sees your tools (the mechanic that governs everything)
+
+Modern MCP clients **defer** tool schemas to save context. The lifecycle in a session:
+
+1. **At rest — names only.** When the server connects, the model gets a flat list of tool
+   **identifiers** (`mcp__<server>__<tool>`) — *names, nothing else*. No parameters, no descriptions,
+   no input/output shapes. For ~50 tools that's a cheap, scannable block.
+2. **Server-instructions block — the one thing loaded upfront.** An MCP server may ship a top-level
+   `instructions` string. This **is** in context from the start, alongside the names. It's the only
+   place you can influence routing *before* the model fetches anything.
+3. **Discovery — the model fetches a schema.** To use a tool, the model runs a tool-search step
+   (exact `select:<name>` or a keyword query) that loads that tool's **full definition** — description
+   prose + parameter JSONSchema — into context. *This is the first moment the model sees how the tool works.*
+4. **Call → result.** The model invokes with arguments; your formatted result returns into context.
+
+Two consequences that drive the whole guide:
+
+- **Names do the *pre-fetch* routing.** The decision "which tool do I even look at?" is made from names
+  (+ the server-instructions block). A name that doesn't signal intent never gets fetched.
+- **Descriptions do the *post-fetch* disambiguation.** They guide correct parameter choice and break
+  ties between near-neighbors — but only *after* the model already chose to fetch. They can't rescue a
+  tool the model never looked at.
+
+> Mental model: the **name** is the search keyword; the **description** is the manual you read after you
+> already pulled the book off the shelf; the **server-instructions block** is the sign on the shelf.
+
+---
+
+## 2. Principle 1 — The name is the routing surface
+
+Because names carry pre-fetch routing, treat them as the most important UX you ship.
+
+- **Front-load the discriminator.** Put the distinguishing token first: `youtube_transcribe`,
+  `facebook_video_transcribe`. The platform/object is what the model scans for.
+- **`platform_object_action` (or `object_action`).** A consistent scheme lets the model *predict*
+  names it hasn't seen and disambiguate neighbors.
+- **Never ship a bare generic verb.** `search` is a trap — search *what*? `search_serp`,
+  `maps_search`, `facebook_ad_search`. The qualifier prevents cross-fetch and sharpens keyword ranking.
+- **Disambiguate near-neighbors on their real axis.** `extract_url` vs `extract_site` vs
+  `map_site_urls` work because the names encode *single page / whole site / URLs-only*. If those blur,
+  the model picks the wrong granularity.
+- **Match the user's vocabulary, or bridge it.** Users say "scrape a page"; if the tool is
+  `extract_url`, that gap is only bridged *after* fetch (via description/`symptomTriggers`). Where a
+  phrasing is common and high-value, make the name or its triggers reflect it.
+- **Mind the namespace stutter.** The client prepends `mcp__`. Naming the server `mcp-scraper` yields
+  `mcp__mcp-scraper__…`. Name it `scraper` → `mcp__scraper__…`. Cosmetic, but it shows in permission
+  prompts/logs, and renaming later is breaking — do it early.
+
+### The decisive test: named tool vs. generic engine
+A user says *"collect ICP info from Reddit."*
+- A **`reddit_research`** tool: the word "reddit" hits the name directly → fetched immediately, one hop.
+- The generic **`browser_*`** agent: matches *none* of the user's words. To land there the model must
+  *infer* "no reddit tool → reddit is a website → drive a browser → navigate → search → read." Several
+  hops, each a chance to mis-route (e.g. grab `search_serp` and Google it instead).
+
+**Lesson:** if there's a phrase users will say that names a platform or intent, you want a **named tool
+catching it** — even if the implementation is a thin wrapper over a generic engine. *The name triggers;
+the engine executes.* A category that merely "points at the browser" serves a human clicking a button;
+it does **not** serve the model that has to *find* the capability from words.
+
+---
+
+## 3. Principle 2 — Split by capability, parameterize the variations
+
+It's tempting to think "names are cheap, so ship hundreds of hyper-specific tools." Don't. Three reasons:
+
+1. **The name list isn't free at scale.** All names load at rest. 50 is fine; 300 is a wall the model
+   must scan and a real token cost. There's a ceiling.
+2. **Search ranking *degrades* with volume.** More near-neighbors → more ambiguous keyword ranking →
+   *higher* wrong-fetch rate. Over-splitting makes the exact failure you're trying to avoid worse.
+3. **It's the combinatorial trap.** `extract_url_screenshot`, `extract_url_branding`,
+   `extract_url_mobile`… is N×M×K variants of one capability. Parameters exist to prevent this.
+
+**The rule:** one tool per *distinct thing a user wants to do*; **parameters** for *variations* of that thing.
+
+The split test:
+> Would a user think of these as **different things to do**, or the **same thing with options**?
+
+- Different platform/object → different tool (`youtube_transcribe` vs `facebook_video_transcribe`).
+- Different granularity / mental model → different tool (`extract_url` vs `extract_site`).
+- A knob on the same action → **parameter** (`screenshot`, `formats`, `maxPages`, `rotateProxies`).
+
+Post-fetch param-learning is cheap and reliable for the model, so you lose almost nothing by using
+params for variations — and you keep the name list scannable and the ranking sharp.
+
+---
+
+## 4. Principle 3 — Descriptions are the post-fetch safety net
+
+Loaded on fetch, the description's job is **correct usage + tie-breaking**, not discovery. Make it earn that:
+
+- **Lead with when-to-use and when-NOT-to-use.** Negative space ("use `extract_url` for one page; use
+  `extract_site` for a whole site") is what separates near-neighbors once both are fetched.
+- **State side effects and cost plainly.** Especially for tools that spend money, open sessions, or
+  post — the model should be able to refuse a wrong pick *before* executing.
+- **Encode parameter intent, bounds, and defaults** that mirror your actual schema, so the model fills
+  the form correctly the first time.
+- **Carry symptom triggers / sample prompts.** Phrases that should route here. These help both the
+  keyword search ranking and the post-fetch confirmation.
+
+Where the net fails (design against this): two tools with **similar names AND overlapping
+descriptions** — the wrong one reads "plausible enough" and the model proceeds. Distinct names +
+explicit negative space prevent it.
+
+---
+
+## 5. Principle 4 — Use the server-instructions block as the routing map
+
+It's the only content loaded *before* fetch besides names, so it's your lever on pre-fetch routing.
+Keep it short and put **cross-tool routing**, not per-tool detail, there:
+
+- A decision map: "single page → `extract_url`; whole site → `extract_site`; just URLs → `map_site_urls`."
+- Setup/auth prerequisites that span tools.
+- How to batch related fetches (clients often let the model load several schemas in one search call —
+  tell it which tools group together).
+
+This is where you catch granularity and family mistakes *before* the model ever fetches the wrong one.
+
+---
+
+## 6. Principle 5 — Two surfaces: human categories vs. AI tools
+
+Your frontend and your MCP tool list are **different products with different optimal shapes.**
+
+- **Humans** benefit from *consolidation*: one "Search" card with a source dropdown
+  (Web / Maps / Ads / YouTube). Fewer, richer entries; group by platform or verb as fits the UI.
+- **The model** benefits from *distinction*: separate, well-named tools (`search_serp`, `maps_search`,
+  `facebook_ad_search`). A single `search(source=…)` tool forces the model to fetch it just to learn the
+  source enum, and the name `search` is ambiguous at discovery.
+
+**Do not let the human IA pressure you into merging MCP tool names.** Map *one frontend card → many
+distinct MCP tools*. Two surfaces, one backend.
+
+Corollary — **a category can exist without a tool, but the AI path can't.** A "Reddit" card can be a
+guided browser flow with no `reddit_*` tool; a human clicks it fine. But the model gets a clean path
+*only* if a named tool exists (see §2). Decide per capability: human-only/occasional → category over the
+generic engine; want strong AI routing → ship the named wrapper.
+
+Platform-first vs verb-first grouping: for a multi-platform tool, **platform-first**
+(YouTube / Facebook / Google Maps / Websites / Browser / Workflows) often wins — it matches how users
+think *and* mirrors `platform_verb` naming. The category is the umbrella (covers search *and*
+transcribe *and* …); the tools inside are the named verbs.
+
+---
+
+## 7. Principle 6 — Tool *results* are context too; keep them lean
+
+Discovery isn't the only context cost — **what you return floods the window if you let it.**
+
+- **Summarize inline, persist the bulk, hand back a reference.** For large outputs, return a concise
+  summary + a saved file / artifact / URL the model can read on demand, not the whole payload.
+- **Truncate with a signpost.** "showing first N of M; full data at `<path>`" beats silently dumping or
+  silently dropping.
+- **Put the full-fidelity data somewhere the model can fetch** (disk file, object storage URL), so the
+  model gets the gist now and the detail only if needed.
+- **Structured + human-readable.** Return both a structured object (for programmatic chaining) and a
+  short readable block (for the model to reason over) — don't make it parse a wall of text.
+
+A tool that returns 200 KB of HTML per call will blow out a multi-step task; one that returns a 2 KB
+summary + a file path scales.
+
+---
+
+## 8. Failure modes → the fix
+
+| Failure | Cause | Fix |
+|---|---|---|
+| **Tool never found** | Name doesn't match user vocabulary | Front-loaded, intent-matching name; `symptomTriggers` |
+| **Wrong tool fetched & believed** | Similar names + overlapping descriptions | Distinct names; explicit negative space; bounds in description |
+| **Routed to generic engine the hard way** | Capability has no named tool | Thin named wrapper over the engine (§2 Reddit) |
+| **Wrong granularity** | `*_url` / `*_site` / `*_map` blur | Encode the axis in the name; routing map in server instructions |
+| **Picked god-tool, wrong sub-mode** | One tool, mega-param surface | Split distinct capabilities into separate tools |
+| **Context blown out** | Tool returns full payload | Summarize + persist + reference (§7) |
+| **Expensive wrong call** | Side-effecting tool, ambiguous name/desc | Clear name; state cost/side-effects up top; confirm before execute |
+| **Namespace stutter / churn** | Server renamed late | Pick the clean server name on day one |
+
+---
+
+## 9. Naming checklist (per tool)
+
+- [ ] Discriminator (platform/object) is **first** in the name.
+- [ ] Follows a consistent `platform_object_action` scheme used across the server.
+- [ ] No bare generic verb; qualifier present.
+- [ ] Distinct from every neighbor on a real axis (object / granularity / platform).
+- [ ] Matches at least one phrase a real user would say (or `symptomTriggers` covers it).
+- [ ] It's a *distinct capability*, not a variation that should be a parameter.
+- [ ] Side effects / cost surfaced in the first lines of the description.
+- [ ] Description leads with when-to-use **and** when-not-to-use (negative space).
+- [ ] Params mirror the real schema (names, bounds, enums, defaults).
+- [ ] Result is lean: summary + reference, not a raw dump.
+
+## 10. Server-level checklist
+
+- [ ] Clean server name (no `mcp__mcp-…` stutter), chosen before launch.
+- [ ] `instructions` block carries a short **cross-tool routing map**, not per-tool detail.
+- [ ] Tool count is at the *capability* grain — not exploded into variants.
+- [ ] Frontend categories map to *multiple distinct* MCP tools (two surfaces).
+- [ ] High-value, user-named capabilities have a **named tool**, even if engine-backed.
+- [ ] Related tools are documented as a batch the model can fetch together.
+
+---
+
+## 11. The one-paragraph version
+
+The model sees only tool **names** until it fetches a schema, so **names carry routing** — front-load
+the discriminator, never ship a bare verb, and give every user-named capability its own named tool
+(even a thin wrapper over a generic engine), because a name that matches the user's words is a one-hop
+hit while a generic engine is a multi-hop inference. Split tools by **distinct capability** and use
+**parameters** for variations — more tools past a point *degrades* selection, it doesn't help.
+**Descriptions** are the post-fetch safety net (when-to-use + when-not, cost, bounds); the
+**server-instructions block** is your only pre-fetch routing lever (put a cross-tool decision map
+there). Keep the **human frontend** (consolidated cards) and the **AI tool list** (distinct names)
+as separate surfaces over one backend. And remember tool **results are context** — return a summary
+plus a reference, never a raw dump.

package/docs/mcp-tool-manifest.generated.json

@@ -1,12 +1,12 @@
 {
-  "generatedAt": "2026-06-23T22:55:40.230Z",
+  "generatedAt": "2026-06-26T17:22:50.447Z",
   "counts": {
-    "main_stdio": 24,
+    "main_stdio": 25,
     "browser_agent_stdio": 20,
-    "combined_stdio": 44,
-    "hosted_http": 26,
+    "combined_stdio": 45,
+    "hosted_http": 27,
     "hosted_only": 2,
-    "unique_public": 46
+    "unique_public": 47
   },
   "surfaces": {
     "main_stdio": [
@@ -15,6 +15,7 @@
       "extract_url",
       "map_site_urls",
       "extract_site",
+      "audit_site",
       "youtube_harvest",
       "youtube_transcribe",
       "facebook_page_intel",
@@ -32,7 +33,7 @@
       "workflow_step",
       "workflow_status",
       "workflow_artifact_read",
-      "rank_tracker_blueprint",
+      "rank_tracker_workflow",
       "credits_info"
     ],
     "browser_agent_stdio": [
@@ -55,7 +56,7 @@
       "browser_replay_annotate",
       "browser_close",
       "browser_list_sessions",
-      "browser_capture_fanout"
+      "query_fanout_workflow"
     ],
     "combined_stdio": [
       "harvest_paa",
@@ -63,6 +64,7 @@
       "extract_url",
       "map_site_urls",
       "extract_site",
+      "audit_site",
       "youtube_harvest",
       "youtube_transcribe",
       "facebook_page_intel",
@@ -80,7 +82,7 @@
       "workflow_step",
       "workflow_status",
       "workflow_artifact_read",
-      "rank_tracker_blueprint",
+      "rank_tracker_workflow",
       "credits_info",
       "browser_open",
       "browser_profile_onboard",
@@ -101,7 +103,7 @@
       "browser_replay_annotate",
       "browser_close",
       "browser_list_sessions",
-      "browser_capture_fanout"
+      "query_fanout_workflow"
     ],
     "hosted_http": [
       "harvest_paa",
@@ -109,6 +111,7 @@
       "extract_url",
       "map_site_urls",
       "extract_site",
+      "audit_site",
       "youtube_harvest",
       "youtube_transcribe",
       "facebook_page_intel",
@@ -126,7 +129,7 @@
       "workflow_step",
       "workflow_status",
       "workflow_artifact_read",
-      "rank_tracker_blueprint",
+      "rank_tracker_workflow",
       "credits_info",
       "capture_serp_snapshot",
       "capture_serp_page_snapshots"
@@ -141,6 +144,7 @@
       "extract_url",
       "map_site_urls",
       "extract_site",
+      "audit_site",
       "youtube_harvest",
       "youtube_transcribe",
       "facebook_page_intel",
@@ -158,7 +162,7 @@
       "workflow_step",
       "workflow_status",
       "workflow_artifact_read",
-      "rank_tracker_blueprint",
+      "rank_tracker_workflow",
       "credits_info",
       "browser_open",
       "browser_profile_onboard",
@@ -179,7 +183,7 @@
       "browser_replay_annotate",
       "browser_close",
       "browser_list_sessions",
-      "browser_capture_fanout",
+      "query_fanout_workflow",
       "capture_serp_snapshot",
       "capture_serp_page_snapshots"
     ]
@@ -299,7 +303,7 @@
       ],
       "formatExamples": [
         "url=\"https://example.com\"",
-        "maxUrls=100 for normal maps, maxUrls=500 for full inventory"
+        "maxUrls=100 for normal maps, up to maxUrls=10000 for a full inventory (maps over 500 URLs write the complete list to a CSV file and return a summary plus the path)"
       ],
       "negativeSpace": "Use extract_url for one known page; use extract_site after mapping when the user wants page content.",
       "producesHandlesFor": [
@@ -322,18 +326,49 @@
       "hasStructuredContent": true,
       "writesLocalFiles": true,
       "symptomTriggers": [
-        "the user wants a multi-page site audit, content crawl, competitor crawl, or schema inventory"
+        "the user wants the page content/text of multiple pages on a site (a content crawl or scrape)"
       ],
       "samplePrompts": [
-        "Extract this site for an SEO audit",
-        "Crawl the top pages of this competitor",
-        "Get titles and schema across the site"
+        "Scrape the content of these pages",
+        "Crawl the top pages of this competitor for their copy",
+        "Get the page content across the site"
       ],
       "formatExamples": [
         "url=\"https://example.com\"",
-        "maxPages=50 for the MCP limit"
+        "maxPages=50 for a normal crawl, up to maxPages=10000 for a full-site bulk scrape (crawls over 25 pages switch to folder mode: each page saved as its own file, response returns a summary plus the folder path)"
       ],
-      "negativeSpace": "Use map_site_urls first when the user needs URL selection; use extract_url for one page.",
+      "negativeSpace": "Returns content only. Use audit_site for a technical SEO audit (issues, link graph, indexability, image weights); use map_site_urls first when the user needs URL selection; use extract_url for one page.",
+      "producesHandlesFor": [
+        "saved-report"
+      ],
+      "consumesHandles": [
+        "public-site-url"
+      ]
+    },
+    {
+      "name": "audit_site",
+      "surfaces": [
+        "main_stdio",
+        "combined_stdio",
+        "hosted_http"
+      ],
+      "tokenBudgetClass": "core",
+      "hasOutputSchema": true,
+      "hasStructuredContent": true,
+      "writesLocalFiles": true,
+      "symptomTriggers": [
+        "the user wants a technical SEO audit, site health check, or Screaming-Frog-style crawl with issues, internal link graph, indexability, heading breakdown, and image sizes/formats"
+      ],
+      "samplePrompts": [
+        "Run a technical SEO audit on this site",
+        "Do a Screaming Frog style crawl of example.com",
+        "Audit this site for SEO issues and oversized images"
+      ],
+      "formatExamples": [
+        "url=\"https://example.com\"",
+        "maxPages=50 for a normal audit, up to 10000 for a full-site audit; always writes a folder of analysis files (report.md, issues.json, pages.jsonl, links.jsonl, link-metrics.jsonl, images.jsonl) plus per-page content and returns a summary plus the folder path"
+      ],
+      "negativeSpace": "Use extract_site when the user only wants page content; use map_site_urls to just list URLs.",
       "producesHandlesFor": [
         "saved-report"
       ],
@@ -861,7 +896,7 @@
       ]
     },
     {
-      "name": "rank_tracker_blueprint",
+      "name": "rank_tracker_workflow",
       "surfaces": [
         "main_stdio",
         "combined_stdio",
@@ -941,7 +976,7 @@
         "browser_screenshot",
         "browser_read",
         "browser_close",
-        "browser_capture_fanout"
+        "query_fanout_workflow"
       ],
       "consumesHandles": []
     },
@@ -1489,7 +1524,7 @@
       "consumesHandles": []
     },
     {
-      "name": "browser_capture_fanout",
+      "name": "query_fanout_workflow",
       "surfaces": [
         "browser_agent_stdio",
         "combined_stdio"