mcp-scraper 0.2.5 → 0.2.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -5
- package/dist/bin/api-server.cjs +399 -2
- package/dist/bin/api-server.cjs.map +1 -1
- package/dist/bin/api-server.js +1 -1
- package/dist/bin/browser-agent-stdio-server.cjs +1 -1
- package/dist/bin/browser-agent-stdio-server.cjs.map +1 -1
- package/dist/bin/browser-agent-stdio-server.js +2 -2
- package/dist/bin/mcp-scraper-combined-stdio-server.cjs +391 -1
- package/dist/bin/mcp-scraper-combined-stdio-server.cjs.map +1 -1
- package/dist/bin/mcp-scraper-combined-stdio-server.js +3 -3
- package/dist/bin/mcp-scraper-install.cjs +3 -2
- package/dist/bin/mcp-scraper-install.cjs.map +1 -1
- package/dist/bin/mcp-scraper-install.js +3 -2
- package/dist/bin/mcp-scraper-install.js.map +1 -1
- package/dist/bin/mcp-stdio-server.cjs +391 -1
- package/dist/bin/mcp-stdio-server.cjs.map +1 -1
- package/dist/bin/mcp-stdio-server.js +2 -2
- package/dist/{chunk-ENI4DM3S.js → chunk-7WB2W6FO.js} +392 -2
- package/dist/chunk-7WB2W6FO.js.map +1 -0
- package/dist/chunk-T6SIPPOQ.js +7 -0
- package/dist/chunk-T6SIPPOQ.js.map +1 -0
- package/dist/{chunk-O36HHUKX.js → chunk-XL4V7VKD.js} +2 -2
- package/dist/{server-Q3WHIS63.js → server-MKCU3M7Y.js} +3 -3
- package/package.json +1 -1
- package/dist/chunk-ENI4DM3S.js.map +0 -1
- package/dist/chunk-JDX57SEC.js +0 -7
- package/dist/chunk-JDX57SEC.js.map +0 -1
- /package/dist/{chunk-O36HHUKX.js.map → chunk-XL4V7VKD.js.map} +0 -0
- /package/dist/{server-Q3WHIS63.js.map → server-MKCU3M7Y.js.map} +0 -0
package/README.md
CHANGED
|
@@ -8,7 +8,7 @@ Use the MCPB Desktop Extension for the branded Claude Desktop install, or use th
|
|
|
8
8
|
|
|
9
9
|
MCP Scraper ships three local stdio entrypoints:
|
|
10
10
|
|
|
11
|
-
- `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook, Maps, directory, and credit tools.
|
|
11
|
+
- `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook, Maps, directory, rank tracker blueprint, and credit tools.
|
|
12
12
|
- `browser-agent` — an agent-controlled live cloud browser with screenshots, clicks, typing, scrolling, live watch URLs, replay links, and MP4 replay download.
|
|
13
13
|
- `mcp-scraper-combined` — one local MCP server that exposes both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
|
|
14
14
|
- `mcp-scraper-install` — a human-facing terminal installer card with the branded ASCII intro and copyable install commands. This command is safe to print because it is not an MCP stdio server.
|
|
@@ -31,7 +31,7 @@ Build the branded one-click bundle:
|
|
|
31
31
|
npm run build:mcpb
|
|
32
32
|
```
|
|
33
33
|
|
|
34
|
-
The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.
|
|
34
|
+
The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.6`, SHA-256 `d316c1e3744d61c84b8fbbf2b155dba7da4062be52d9024d9fa7adde04c5b73c`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
|
|
35
35
|
|
|
36
36
|
The MCPB install exposes the same web-intelligence tools as `mcp-scraper` plus all `browser_*` tools from `browser-agent` through one server.
|
|
37
37
|
|
|
@@ -108,6 +108,7 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
|
|
|
108
108
|
- `maps_search` — search Google Maps for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. `maxResults` defaults to 10 and is capped at 50.
|
|
109
109
|
- `maps_place_intel` — hydrate one known/named Google Maps business with profile details and optional reviews. Use after `maps_search` when a selected candidate needs full details.
|
|
110
110
|
- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus Google Maps searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups. It captures Maps star ratings from list cards, not profile review counts.
|
|
111
|
+
- `rank_tracker_blueprint` — generate a database schema, cron/heartbeat plan, ingestion workflow, metrics list, and implementation prompt for building rank trackers. It has modes for Maps rankings via `directory_workflow`/`maps_search`, organic rankings via `search_serp`, AI Overview citation tracking, and PAA source presence tracking. This is a local planning tool and does not spend credits.
|
|
111
112
|
- `credits_info`
|
|
112
113
|
|
|
113
114
|
### Browser-agent tools
|
|
@@ -131,9 +132,9 @@ For US local SERP tools (`harvest_paa` and `search_serp`), keep `proxyMode` at t
|
|
|
131
132
|
|
|
132
133
|
The MCPB bundle and `mcp-scraper-combined` expose both sections through one local MCP server. The split `mcp-scraper` entrypoint exposes only the web-intelligence tools, and the split `browser-agent` entrypoint exposes only the browser-agent tools.
|
|
133
134
|
|
|
134
|
-
Chaining tools (`maps_search`, `map_site_urls`, `youtube_harvest`, `facebook_ad_search`, `facebook_page_intel`) advertise an `outputSchema` and return `structuredContent` with the IDs
|
|
135
|
+
Chaining and planning tools (`maps_search`, `directory_workflow`, `rank_tracker_blueprint`, `map_site_urls`, `youtube_harvest`, `facebook_ad_search`, `facebook_page_intel`) advertise an `outputSchema` and return `structuredContent` with the IDs, URLs, CSV paths, or blueprint fields needed by the next step. All tools carry MCP annotations (`readOnlyHint: true`; live-web tools use `openWorldHint: true`, while local planning and credit tools are closed-world/idempotent).
|
|
135
136
|
|
|
136
|
-
The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the
|
|
137
|
+
The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 15 web-intelligence tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (17 total). Browser-agent tools are local stdio tools backed by the REST API under `https://mcpscraper.dev/agent/*`.
|
|
137
138
|
|
|
138
139
|
## Resources
|
|
139
140
|
|
|
@@ -173,7 +174,7 @@ npm update -g mcp-scraper
|
|
|
173
174
|
npm install mcp-scraper@latest
|
|
174
175
|
```
|
|
175
176
|
|
|
176
|
-
Users who do not update can keep using the tools their local package already advertises, but they will not see newly added local stdio tools, schemas, or AI-facing descriptions. For example, a client running an older local package cannot call `directory_workflow
|
|
177
|
+
Users who do not update can keep using the tools their local package already advertises, but they will not see newly added local stdio tools, schemas, or AI-facing descriptions. For example, a client running an older local package cannot call `rank_tracker_blueprint`, `directory_workflow`, or browser tools through stdio even if the hosted API already supports adjacent endpoints. Users who configured only `mcp-scraper` must switch to `mcp-scraper-combined` or add `browser-agent` separately; MCP clients do not auto-create a second server entry from an existing config.
|
|
177
178
|
|
|
178
179
|
## Branded One-Click Installs
|
|
179
180
|
|
package/dist/bin/api-server.cjs
CHANGED
|
@@ -17082,12 +17082,12 @@ var PACKAGE_VERSION;
|
|
|
17082
17082
|
var init_version = __esm({
|
|
17083
17083
|
"src/version.ts"() {
|
|
17084
17084
|
"use strict";
|
|
17085
|
-
PACKAGE_VERSION = "0.2.
|
|
17085
|
+
PACKAGE_VERSION = "0.2.6";
|
|
17086
17086
|
}
|
|
17087
17087
|
});
|
|
17088
17088
|
|
|
17089
17089
|
// src/mcp/mcp-tool-schemas.ts
|
|
17090
|
-
var import_zod20, HarvestPaaInputSchema, ExtractUrlInputSchema, MapSiteUrlsInputSchema, ExtractSiteInputSchema, YoutubeHarvestInputSchema, YoutubeTranscribeInputSchema, FacebookPageIntelInputSchema, FacebookAdSearchInputSchema, FacebookAdTranscribeInputSchema, MapsPlaceIntelInputSchema, MapsSearchInputSchema, DirectoryWorkflowInputSchema, NullableString, MapsSearchOutputSchema, DirectoryMapsBusinessOutput, DirectoryWorkflowOutputSchema, OrganicResultOutput, AiOverviewOutput, EntityIdsOutput, HarvestPaaOutputSchema, SearchSerpOutputSchema, ExtractUrlOutputSchema, ExtractSiteOutputSchema, MapsPlaceIntelOutputSchema, CreditsInfoOutputSchema, MapSiteUrlsOutputSchema, YoutubeHarvestOutputSchema, FacebookAdSearchOutputSchema, FacebookPageIntelOutputSchema, CreditsInfoInputSchema, SearchSerpInputSchema, CaptureSerpSnapshotInputSchema, ScreenshotInputSchema, CaptureSerpPageSnapshotsInputSchema;
|
|
17090
|
+
var import_zod20, HarvestPaaInputSchema, ExtractUrlInputSchema, MapSiteUrlsInputSchema, ExtractSiteInputSchema, YoutubeHarvestInputSchema, YoutubeTranscribeInputSchema, FacebookPageIntelInputSchema, FacebookAdSearchInputSchema, FacebookAdTranscribeInputSchema, MapsPlaceIntelInputSchema, MapsSearchInputSchema, DirectoryWorkflowInputSchema, RankTrackerModeSchema, RankTrackerBlueprintInputSchema, NullableString, MapsSearchOutputSchema, DirectoryMapsBusinessOutput, DirectoryWorkflowOutputSchema, RankTrackerToolPlanOutput, RankTrackerTableOutput, RankTrackerCronJobOutput, RankTrackerBlueprintOutputSchema, OrganicResultOutput, AiOverviewOutput, EntityIdsOutput, HarvestPaaOutputSchema, SearchSerpOutputSchema, ExtractUrlOutputSchema, ExtractSiteOutputSchema, MapsPlaceIntelOutputSchema, CreditsInfoOutputSchema, MapSiteUrlsOutputSchema, YoutubeHarvestOutputSchema, FacebookAdSearchOutputSchema, FacebookPageIntelOutputSchema, CreditsInfoInputSchema, SearchSerpInputSchema, CaptureSerpSnapshotInputSchema, ScreenshotInputSchema, CaptureSerpPageSnapshotsInputSchema;
|
|
17091
17091
|
var init_mcp_tool_schemas = __esm({
|
|
17092
17092
|
"src/mcp/mcp-tool-schemas.ts"() {
|
|
17093
17093
|
"use strict";
|
|
@@ -17177,6 +17177,24 @@ var init_mcp_tool_schemas = __esm({
|
|
|
17177
17177
|
proxyZip: import_zod20.z.string().regex(/^\d{5}$/).optional().describe("Optional ZIP override for proxy targeting. Normally omit it so each city can use its Lead Magician ZIP group or city/state location."),
|
|
17178
17178
|
debug: import_zod20.z.boolean().default(false).describe("Include sanitized browser/proxy diagnostics in each Maps browser session when supported.")
|
|
17179
17179
|
};
|
|
17180
|
+
RankTrackerModeSchema = import_zod20.z.enum(["maps", "organic", "ai_overview", "paa"]);
|
|
17181
|
+
RankTrackerBlueprintInputSchema = {
|
|
17182
|
+
projectName: import_zod20.z.string().min(1).optional().describe("Optional name for the rank tracker project, client, or campaign."),
|
|
17183
|
+
targetDomain: import_zod20.z.string().min(1).optional().describe("Primary domain to track in organic results, AI Overview citations, and PAA sources, e.g. example.com."),
|
|
17184
|
+
targetBusinessName: import_zod20.z.string().min(1).optional().describe("Primary Google Business Profile or brand/business name to match in Maps results. Required for reliable Maps rank tracking."),
|
|
17185
|
+
trackingModes: import_zod20.z.array(RankTrackerModeSchema).min(1).max(4).default(["maps", "organic", "ai_overview", "paa"]).describe("Rank tracker surfaces to build. maps uses directory_workflow/maps_search. organic uses search_serp. ai_overview uses search_serp/harvest_paa. paa uses harvest_paa source presence."),
|
|
17186
|
+
keywords: import_zod20.z.array(import_zod20.z.string().min(1)).max(200).default([]).describe("Seed keywords or service queries to track. Leave empty when the downstream AI should create the keyword table from user input."),
|
|
17187
|
+
locations: import_zod20.z.array(import_zod20.z.string().min(1)).max(100).default([]).describe("Markets, cities, ZIPs, or service areas to track. Use city/state strings like Denver, CO for localized SERP and Maps checks."),
|
|
17188
|
+
competitors: import_zod20.z.array(import_zod20.z.string().min(1)).max(100).default([]).describe("Optional competitor domains or business names to persist as comparison targets."),
|
|
17189
|
+
database: import_zod20.z.enum(["postgres", "neon", "supabase", "sqlite", "mysql"]).default("postgres").describe("Database family the downstream AI should target when generating migrations."),
|
|
17190
|
+
scheduleCadence: import_zod20.z.enum(["daily", "weekly", "monthly", "custom"]).default("weekly").describe("Default recurring rank check cadence for the generated cron/heartbeat plan."),
|
|
17191
|
+
customCron: import_zod20.z.string().min(1).optional().describe("Cron expression to use when scheduleCadence is custom."),
|
|
17192
|
+
timezone: import_zod20.z.string().min(1).default("UTC").describe("IANA timezone for scheduled rank checks, e.g. America/Denver."),
|
|
17193
|
+
includeCron: import_zod20.z.boolean().default(true).describe("Include a cron or heartbeat worker plan. Keep true for production rank trackers."),
|
|
17194
|
+
includeDashboard: import_zod20.z.boolean().default(true).describe("Include dashboard/reporting requirements in the generated prompt."),
|
|
17195
|
+
includeAlerts: import_zod20.z.boolean().default(true).describe("Include alert rules for rank movement and SERP feature gains/losses."),
|
|
17196
|
+
notes: import_zod20.z.string().max(4e3).optional().describe("Extra product, client, stack, or hosting requirements to include in the implementation prompt.")
|
|
17197
|
+
};
|
|
17180
17198
|
NullableString = import_zod20.z.string().nullable();
|
|
17181
17199
|
MapsSearchOutputSchema = {
|
|
17182
17200
|
query: import_zod20.z.string(),
|
|
@@ -17252,6 +17270,39 @@ var init_mcp_tool_schemas = __esm({
|
|
|
17252
17270
|
})),
|
|
17253
17271
|
durationMs: import_zod20.z.number().int().min(0)
|
|
17254
17272
|
};
|
|
17273
|
+
RankTrackerToolPlanOutput = import_zod20.z.object({
|
|
17274
|
+
tool: import_zod20.z.string(),
|
|
17275
|
+
purpose: import_zod20.z.string()
|
|
17276
|
+
});
|
|
17277
|
+
RankTrackerTableOutput = import_zod20.z.object({
|
|
17278
|
+
name: import_zod20.z.string(),
|
|
17279
|
+
purpose: import_zod20.z.string(),
|
|
17280
|
+
keyColumns: import_zod20.z.array(import_zod20.z.string())
|
|
17281
|
+
});
|
|
17282
|
+
RankTrackerCronJobOutput = import_zod20.z.object({
|
|
17283
|
+
name: import_zod20.z.string(),
|
|
17284
|
+
purpose: import_zod20.z.string(),
|
|
17285
|
+
modes: import_zod20.z.array(RankTrackerModeSchema),
|
|
17286
|
+
recommendedTools: import_zod20.z.array(import_zod20.z.string())
|
|
17287
|
+
});
|
|
17288
|
+
RankTrackerBlueprintOutputSchema = {
|
|
17289
|
+
projectName: import_zod20.z.string(),
|
|
17290
|
+
targetDomain: NullableString,
|
|
17291
|
+
targetBusinessName: NullableString,
|
|
17292
|
+
trackingModes: import_zod20.z.array(RankTrackerModeSchema),
|
|
17293
|
+
database: import_zod20.z.string(),
|
|
17294
|
+
recommendedTools: import_zod20.z.array(RankTrackerToolPlanOutput),
|
|
17295
|
+
tables: import_zod20.z.array(RankTrackerTableOutput),
|
|
17296
|
+
cron: import_zod20.z.object({
|
|
17297
|
+
enabled: import_zod20.z.boolean(),
|
|
17298
|
+
cadence: import_zod20.z.string(),
|
|
17299
|
+
expression: import_zod20.z.string(),
|
|
17300
|
+
timezone: import_zod20.z.string(),
|
|
17301
|
+
jobs: import_zod20.z.array(RankTrackerCronJobOutput)
|
|
17302
|
+
}),
|
|
17303
|
+
metrics: import_zod20.z.array(import_zod20.z.string()),
|
|
17304
|
+
implementationPrompt: import_zod20.z.string()
|
|
17305
|
+
};
|
|
17255
17306
|
OrganicResultOutput = import_zod20.z.object({
|
|
17256
17307
|
position: import_zod20.z.number().int(),
|
|
17257
17308
|
title: import_zod20.z.string(),
|
|
@@ -17468,6 +17519,335 @@ var init_mcp_tool_schemas = __esm({
|
|
|
17468
17519
|
}
|
|
17469
17520
|
});
|
|
17470
17521
|
|
|
17522
|
+
// src/mcp/rank-tracker-blueprint.ts
|
|
17523
|
+
function normalizeDomain(value) {
|
|
17524
|
+
const trimmed = value?.trim();
|
|
17525
|
+
if (!trimmed) return null;
|
|
17526
|
+
try {
|
|
17527
|
+
const url = new URL(trimmed.includes("://") ? trimmed : `https://${trimmed}`);
|
|
17528
|
+
return url.hostname.replace(/^www\./, "").toLowerCase();
|
|
17529
|
+
} catch {
|
|
17530
|
+
return trimmed.replace(/^https?:\/\//, "").replace(/^www\./, "").split("/")[0]?.toLowerCase() || trimmed;
|
|
17531
|
+
}
|
|
17532
|
+
}
|
|
17533
|
+
function uniqueModes(input) {
|
|
17534
|
+
const modes = input?.length ? input : DEFAULT_MODES;
|
|
17535
|
+
return [...new Set(modes)];
|
|
17536
|
+
}
|
|
17537
|
+
function cronExpression(cadence, customCron) {
|
|
17538
|
+
if (cadence === "daily") return "15 6 * * *";
|
|
17539
|
+
if (cadence === "monthly") return "15 6 1 * *";
|
|
17540
|
+
if (cadence === "custom") return customCron?.trim() || "CUSTOM_CRON_EXPRESSION_REQUIRED";
|
|
17541
|
+
return "15 6 * * 1";
|
|
17542
|
+
}
|
|
17543
|
+
function includes(modes, mode) {
|
|
17544
|
+
return modes.includes(mode);
|
|
17545
|
+
}
|
|
17546
|
+
function buildToolPlan(modes) {
|
|
17547
|
+
const plans = [];
|
|
17548
|
+
const push = (tool, purpose) => {
|
|
17549
|
+
if (!plans.some((plan) => plan.tool === tool)) plans.push({ tool, purpose });
|
|
17550
|
+
};
|
|
17551
|
+
push("credits_info", "Check account balance and rates before scheduling recurring rank checks.");
|
|
17552
|
+
if (includes(modes, "maps")) {
|
|
17553
|
+
push("directory_workflow", "Seed or refresh city-by-city Maps datasets when building local market coverage.");
|
|
17554
|
+
push("maps_search", "Run recurring Google Maps category checks for each keyword and location.");
|
|
17555
|
+
push("maps_place_intel", "Hydrate selected target or competitor GBP details when profile-level evidence is needed.");
|
|
17556
|
+
}
|
|
17557
|
+
if (includes(modes, "organic")) {
|
|
17558
|
+
push("search_serp", "Capture localized organic rankings, local pack data, and AI Overview status without PAA expansion.");
|
|
17559
|
+
}
|
|
17560
|
+
if (includes(modes, "ai_overview")) {
|
|
17561
|
+
push("search_serp", "Capture quick AI Overview detection and organic context for each tracked query.");
|
|
17562
|
+
push("harvest_paa", "Use when AI Overview text, PAA context, or source-level evidence needs deeper capture.");
|
|
17563
|
+
}
|
|
17564
|
+
if (includes(modes, "paa")) {
|
|
17565
|
+
push("harvest_paa", "Capture People Also Ask questions, answers, and source sites for PAA presence tracking.");
|
|
17566
|
+
}
|
|
17567
|
+
return plans;
|
|
17568
|
+
}
|
|
17569
|
+
function buildTables(modes, includeDashboard, includeAlerts, hasCompetitors) {
|
|
17570
|
+
const tables = [
|
|
17571
|
+
{
|
|
17572
|
+
name: "rank_tracker_projects",
|
|
17573
|
+
purpose: "One row per client, campaign, or tracked property.",
|
|
17574
|
+
keyColumns: ["id", "name", "target_domain", "target_business_name", "database_mode", "timezone", "created_at"]
|
|
17575
|
+
},
|
|
17576
|
+
{
|
|
17577
|
+
name: "rank_tracker_targets",
|
|
17578
|
+
purpose: "Normalized target identities and match rules for domains, URLs, business names, CIDs, and competitors.",
|
|
17579
|
+
keyColumns: ["id", "project_id", "kind", "value", "display_name", "match_rules_json", "active"]
|
|
17580
|
+
},
|
|
17581
|
+
{
|
|
17582
|
+
name: "rank_tracker_keywords",
|
|
17583
|
+
purpose: "Tracked keywords or service queries, separated from location so calls can localize correctly.",
|
|
17584
|
+
keyColumns: ["id", "project_id", "keyword", "intent", "tags_json", "active"]
|
|
17585
|
+
},
|
|
17586
|
+
{
|
|
17587
|
+
name: "rank_tracker_locations",
|
|
17588
|
+
purpose: "Markets, cities, ZIPs, country/language settings, and proxy targeting hints.",
|
|
17589
|
+
keyColumns: ["id", "project_id", "label", "gl", "hl", "proxy_zip", "device", "active"]
|
|
17590
|
+
},
|
|
17591
|
+
{
|
|
17592
|
+
name: "rank_tracker_runs",
|
|
17593
|
+
purpose: "Heartbeat table for every scheduled or manual rank check batch.",
|
|
17594
|
+
keyColumns: ["id", "project_id", "scheduled_for", "started_at", "completed_at", "status", "idempotency_key", "error_message"]
|
|
17595
|
+
}
|
|
17596
|
+
];
|
|
17597
|
+
if (includes(modes, "organic")) {
|
|
17598
|
+
tables.push({
|
|
17599
|
+
name: "rank_tracker_organic_results",
|
|
17600
|
+
purpose: "SERP organic result rows from search_serp, including target-domain matches and competitor rows.",
|
|
17601
|
+
keyColumns: ["id", "run_id", "keyword_id", "location_id", "device", "position", "title", "url", "domain", "is_target"]
|
|
17602
|
+
});
|
|
17603
|
+
}
|
|
17604
|
+
if (includes(modes, "maps")) {
|
|
17605
|
+
tables.push({
|
|
17606
|
+
name: "rank_tracker_maps_results",
|
|
17607
|
+
purpose: "Google Maps result rows from maps_search and directory_workflow, including GBP identity fields.",
|
|
17608
|
+
keyColumns: ["id", "run_id", "keyword_id", "location_id", "position", "business_name", "place_url", "cid", "website_url", "rating", "review_count", "is_target"]
|
|
17609
|
+
});
|
|
17610
|
+
tables.push({
|
|
17611
|
+
name: "rank_tracker_market_seeds",
|
|
17612
|
+
purpose: "Directory workflow city, population, ZIP-group, and market seed records used to build local tracking coverage.",
|
|
17613
|
+
keyColumns: ["id", "project_id", "query", "city", "state", "population", "zip_group_json", "last_seeded_run_id"]
|
|
17614
|
+
});
|
|
17615
|
+
}
|
|
17616
|
+
if (includes(modes, "ai_overview")) {
|
|
17617
|
+
tables.push({
|
|
17618
|
+
name: "rank_tracker_ai_overviews",
|
|
17619
|
+
purpose: "AI Overview detection, text snapshot, citation domains, and target citation status per query/location.",
|
|
17620
|
+
keyColumns: ["id", "run_id", "keyword_id", "location_id", "detected", "target_cited", "cited_domains_json", "overview_text"]
|
|
17621
|
+
});
|
|
17622
|
+
}
|
|
17623
|
+
if (includes(modes, "paa")) {
|
|
17624
|
+
tables.push({
|
|
17625
|
+
name: "rank_tracker_paa_sources",
|
|
17626
|
+
purpose: "People Also Ask questions, answers, and source sites, with target source presence flags.",
|
|
17627
|
+
keyColumns: ["id", "run_id", "keyword_id", "location_id", "question", "answer", "source_title", "source_site", "target_present"]
|
|
17628
|
+
});
|
|
17629
|
+
}
|
|
17630
|
+
if (hasCompetitors) {
|
|
17631
|
+
tables.push({
|
|
17632
|
+
name: "rank_tracker_competitor_snapshots",
|
|
17633
|
+
purpose: "Per-run competitor visibility summaries across organic, Maps, AI Overview, and PAA surfaces.",
|
|
17634
|
+
keyColumns: ["id", "run_id", "target_id", "surface", "best_position", "presence_count", "share_of_surface"]
|
|
17635
|
+
});
|
|
17636
|
+
}
|
|
17637
|
+
if (includeDashboard) {
|
|
17638
|
+
tables.push({
|
|
17639
|
+
name: "rank_tracker_daily_metrics",
|
|
17640
|
+
purpose: "Rollup table for dashboard charts, trend lines, and share-of-visibility summaries.",
|
|
17641
|
+
keyColumns: ["id", "project_id", "metric_date", "keyword_id", "location_id", "surface", "metric_name", "metric_value"]
|
|
17642
|
+
});
|
|
17643
|
+
}
|
|
17644
|
+
if (includeAlerts) {
|
|
17645
|
+
tables.push({
|
|
17646
|
+
name: "rank_tracker_alert_events",
|
|
17647
|
+
purpose: "Deduplicated gain/loss events for rank movement, Maps top 3, AI Overview citations, and PAA source presence.",
|
|
17648
|
+
keyColumns: ["id", "project_id", "run_id", "surface", "severity", "event_key", "message", "created_at", "acknowledged_at"]
|
|
17649
|
+
});
|
|
17650
|
+
}
|
|
17651
|
+
return tables;
|
|
17652
|
+
}
|
|
17653
|
+
function buildCronJobs(modes, includeDashboard, includeAlerts) {
|
|
17654
|
+
const jobs = [
|
|
17655
|
+
{
|
|
17656
|
+
name: "rank-tracker-dispatch",
|
|
17657
|
+
purpose: "Find due keyword/location/mode combinations, create a rank_tracker_runs row, and enqueue idempotent work items.",
|
|
17658
|
+
modes,
|
|
17659
|
+
recommendedTools: ["credits_info"]
|
|
17660
|
+
}
|
|
17661
|
+
];
|
|
17662
|
+
if (includes(modes, "maps")) {
|
|
17663
|
+
jobs.push({
|
|
17664
|
+
name: "maps-rank-check",
|
|
17665
|
+
purpose: "Run maps_search per keyword/location and optionally directory_workflow for market seed refreshes.",
|
|
17666
|
+
modes: ["maps"],
|
|
17667
|
+
recommendedTools: ["maps_search", "directory_workflow", "maps_place_intel"]
|
|
17668
|
+
});
|
|
17669
|
+
}
|
|
17670
|
+
if (includes(modes, "organic") || includes(modes, "ai_overview")) {
|
|
17671
|
+
jobs.push({
|
|
17672
|
+
name: "serp-rank-check",
|
|
17673
|
+
purpose: "Run search_serp per keyword/location/device and persist organic ranks, local pack, and AI Overview status.",
|
|
17674
|
+
modes: modes.filter((mode) => mode === "organic" || mode === "ai_overview"),
|
|
17675
|
+
recommendedTools: ["search_serp"]
|
|
17676
|
+
});
|
|
17677
|
+
}
|
|
17678
|
+
if (includes(modes, "paa") || includes(modes, "ai_overview")) {
|
|
17679
|
+
jobs.push({
|
|
17680
|
+
name: "paa-ai-evidence-check",
|
|
17681
|
+
purpose: "Run harvest_paa for PAA source presence and deeper AI Overview/PAA evidence where needed.",
|
|
17682
|
+
modes: modes.filter((mode) => mode === "paa" || mode === "ai_overview"),
|
|
17683
|
+
recommendedTools: ["harvest_paa"]
|
|
17684
|
+
});
|
|
17685
|
+
}
|
|
17686
|
+
if (includeDashboard) {
|
|
17687
|
+
jobs.push({
|
|
17688
|
+
name: "rank-tracker-rollups",
|
|
17689
|
+
purpose: "Aggregate raw rows into daily metrics for charts, comparison tables, and trend lines.",
|
|
17690
|
+
modes,
|
|
17691
|
+
recommendedTools: []
|
|
17692
|
+
});
|
|
17693
|
+
}
|
|
17694
|
+
if (includeAlerts) {
|
|
17695
|
+
jobs.push({
|
|
17696
|
+
name: "rank-tracker-alerts",
|
|
17697
|
+
purpose: "Compare latest run against prior snapshots and emit deduplicated visibility gain/loss events.",
|
|
17698
|
+
modes,
|
|
17699
|
+
recommendedTools: []
|
|
17700
|
+
});
|
|
17701
|
+
}
|
|
17702
|
+
return jobs;
|
|
17703
|
+
}
|
|
17704
|
+
function buildMetrics(modes, includeDashboard, includeAlerts) {
|
|
17705
|
+
const metrics = ["run_success_rate", "last_successful_run_at", "stale_keyword_location_count"];
|
|
17706
|
+
if (includes(modes, "maps")) {
|
|
17707
|
+
metrics.push("maps_best_position", "maps_top_3_presence", "maps_target_found", "maps_competitor_count_above_target");
|
|
17708
|
+
}
|
|
17709
|
+
if (includes(modes, "organic")) {
|
|
17710
|
+
metrics.push("organic_best_position", "organic_top_3_presence", "organic_top_10_presence", "organic_best_url");
|
|
17711
|
+
}
|
|
17712
|
+
if (includes(modes, "ai_overview")) {
|
|
17713
|
+
metrics.push("ai_overview_detected", "target_ai_overview_cited", "ai_overview_citation_domain_count");
|
|
17714
|
+
}
|
|
17715
|
+
if (includes(modes, "paa")) {
|
|
17716
|
+
metrics.push("paa_question_count", "target_paa_source_count", "paa_presence_rate", "paa_source_domains");
|
|
17717
|
+
}
|
|
17718
|
+
if (includeDashboard) metrics.push("share_of_visibility", "position_delta_7d", "position_delta_30d");
|
|
17719
|
+
if (includeAlerts) metrics.push("visibility_gain_events", "visibility_loss_events");
|
|
17720
|
+
return metrics;
|
|
17721
|
+
}
|
|
17722
|
+
function listOrPlaceholder(values, fallback) {
|
|
17723
|
+
const clean2 = values?.map((value) => value.trim()).filter(Boolean) ?? [];
|
|
17724
|
+
if (!clean2.length) return fallback;
|
|
17725
|
+
return clean2.map((value) => `- ${value}`).join("\n");
|
|
17726
|
+
}
|
|
17727
|
+
function buildPrompt(input, modes, tools, tables, jobs, metrics, expression, targetDomain, targetBusinessName) {
|
|
17728
|
+
const keywords = listOrPlaceholder(input.keywords, "- TODO: add tracked keywords");
|
|
17729
|
+
const locations = listOrPlaceholder(input.locations, "- TODO: add tracked markets/locations");
|
|
17730
|
+
const competitors = listOrPlaceholder(input.competitors, "- Optional: add competitor domains or GBP names");
|
|
17731
|
+
const notes = input.notes?.trim() || "No extra implementation notes.";
|
|
17732
|
+
return [
|
|
17733
|
+
"You are building a production rank tracker powered by MCP Scraper. Create the database migrations, scheduled worker, ingestion functions, and dashboard/query layer described below.",
|
|
17734
|
+
"",
|
|
17735
|
+
`Project: ${input.projectName || "MCP Scraper Rank Tracker"}`,
|
|
17736
|
+
`Target domain: ${targetDomain || "TODO_TARGET_DOMAIN"}`,
|
|
17737
|
+
`Target business/GBP name: ${targetBusinessName || "TODO_TARGET_BUSINESS_NAME"}`,
|
|
17738
|
+
`Database target: ${input.database || "postgres"}`,
|
|
17739
|
+
`Tracking modes: ${modes.join(", ")}`,
|
|
17740
|
+
"",
|
|
17741
|
+
"Seed keywords:",
|
|
17742
|
+
keywords,
|
|
17743
|
+
"",
|
|
17744
|
+
"Seed locations:",
|
|
17745
|
+
locations,
|
|
17746
|
+
"",
|
|
17747
|
+
"Competitors:",
|
|
17748
|
+
competitors,
|
|
17749
|
+
"",
|
|
17750
|
+
"MCP Scraper calls to use:",
|
|
17751
|
+
tools.map((tool) => `- ${tool.tool}: ${tool.purpose}`).join("\n"),
|
|
17752
|
+
"",
|
|
17753
|
+
"Database tables to create:",
|
|
17754
|
+
tables.map((table) => `- ${table.name}: ${table.keyColumns.join(", ")}`).join("\n"),
|
|
17755
|
+
"",
|
|
17756
|
+
"Scheduler and heartbeat requirements:",
|
|
17757
|
+
`- Use cron expression "${expression}" in timezone "${input.timezone || "UTC"}" unless the user changes cadence.`,
|
|
17758
|
+
"- Every scheduled batch must insert or update rank_tracker_runs before calling MCP tools.",
|
|
17759
|
+
"- Use idempotency_key = project_id + keyword_id + location_id + mode + device + scheduled_date.",
|
|
17760
|
+
"- Mark runs as running, succeeded, failed, or skipped; persist error_message on failures.",
|
|
17761
|
+
"- Run MCP calls sequentially by default because accounts have one concurrency slot unless the user bought more.",
|
|
17762
|
+
"- Retry retryable upstream failures with backoff, but do not duplicate rows for the same idempotency key.",
|
|
17763
|
+
"",
|
|
17764
|
+
"Mode-specific ingestion rules:",
|
|
17765
|
+
"- maps: Use directory_workflow to seed/refresh market coverage and maps_search for recurring keyword/location rank checks. Match the target by targetBusinessName, website domain, CID, and place URL when available.",
|
|
17766
|
+
"- organic: Use search_serp and persist every organic result row. Compute the best targetDomain position and best ranking URL per keyword/location/device.",
|
|
17767
|
+
"- ai_overview: Use search_serp for quick detection and harvest_paa when deeper text/source evidence is needed. Track detected, overview_text, cited domains, and whether targetDomain is cited.",
|
|
17768
|
+
"- paa: Use harvest_paa and persist each question/source pair. Track whether sourceSite or answer/source evidence matches targetDomain, then compute PAA presence rate.",
|
|
17769
|
+
"",
|
|
17770
|
+
"Metrics to expose:",
|
|
17771
|
+
metrics.map((metric) => `- ${metric}`).join("\n"),
|
|
17772
|
+
"",
|
|
17773
|
+
"Alert requirements:",
|
|
17774
|
+
input.includeAlerts ? "- Emit deduplicated alert_events when target gains/loses Maps top 3, organic top 10, AI Overview citation, or PAA source presence." : "- Alerts are out of scope for this build unless the user asks for them.",
|
|
17775
|
+
"",
|
|
17776
|
+
"Dashboard requirements:",
|
|
17777
|
+
input.includeDashboard ? "- Build views for latest position by keyword/location, trend deltas, competitors above target, AI Overview citations, and PAA source wins/losses." : "- Dashboard is out of scope for this build unless the user asks for it.",
|
|
17778
|
+
"",
|
|
17779
|
+
"Extra notes:",
|
|
17780
|
+
notes
|
|
17781
|
+
].join("\n");
|
|
17782
|
+
}
|
|
17783
|
+
function buildRankTrackerBlueprint(input) {
|
|
17784
|
+
const trackingModes = uniqueModes(input.trackingModes);
|
|
17785
|
+
const targetDomain = normalizeDomain(input.targetDomain);
|
|
17786
|
+
const targetBusinessName = input.targetBusinessName?.trim() || null;
|
|
17787
|
+
const projectName = input.projectName?.trim() || "MCP Scraper Rank Tracker";
|
|
17788
|
+
const database = input.database || "postgres";
|
|
17789
|
+
const expression = cronExpression(input.scheduleCadence || "weekly", input.customCron);
|
|
17790
|
+
const tools = buildToolPlan(trackingModes);
|
|
17791
|
+
const tables = buildTables(trackingModes, input.includeDashboard !== false, input.includeAlerts !== false, Boolean(input.competitors?.length));
|
|
17792
|
+
const jobs = input.includeCron === false ? [] : buildCronJobs(trackingModes, input.includeDashboard !== false, input.includeAlerts !== false);
|
|
17793
|
+
const metrics = buildMetrics(trackingModes, input.includeDashboard !== false, input.includeAlerts !== false);
|
|
17794
|
+
const implementationPrompt = buildPrompt(input, trackingModes, tools, tables, jobs, metrics, expression, targetDomain, targetBusinessName);
|
|
17795
|
+
const blueprint = {
|
|
17796
|
+
projectName,
|
|
17797
|
+
targetDomain,
|
|
17798
|
+
targetBusinessName,
|
|
17799
|
+
trackingModes,
|
|
17800
|
+
database,
|
|
17801
|
+
recommendedTools: tools,
|
|
17802
|
+
tables,
|
|
17803
|
+
cron: {
|
|
17804
|
+
enabled: input.includeCron !== false,
|
|
17805
|
+
cadence: input.scheduleCadence || "weekly",
|
|
17806
|
+
expression: input.includeCron === false ? "disabled" : expression,
|
|
17807
|
+
timezone: input.timezone || "UTC",
|
|
17808
|
+
jobs
|
|
17809
|
+
},
|
|
17810
|
+
metrics,
|
|
17811
|
+
implementationPrompt
|
|
17812
|
+
};
|
|
17813
|
+
const text = [
|
|
17814
|
+
`# ${projectName}`,
|
|
17815
|
+
"",
|
|
17816
|
+
"Rank tracker build blueprint generated from MCP Scraper tool capabilities.",
|
|
17817
|
+
"",
|
|
17818
|
+
"## Modes",
|
|
17819
|
+
trackingModes.map((mode) => `- ${mode}`).join("\n"),
|
|
17820
|
+
"",
|
|
17821
|
+
"## Recommended MCP Tools",
|
|
17822
|
+
tools.map((tool) => `- \`${tool.tool}\` - ${tool.purpose}`).join("\n"),
|
|
17823
|
+
"",
|
|
17824
|
+
"## Database Tables",
|
|
17825
|
+
tables.map((table) => `- \`${table.name}\` - ${table.purpose}`).join("\n"),
|
|
17826
|
+
"",
|
|
17827
|
+
"## Cron / Heartbeat",
|
|
17828
|
+
blueprint.cron.enabled ? `- Cadence: ${blueprint.cron.cadence}
|
|
17829
|
+
- Cron: \`${blueprint.cron.expression}\`
|
|
17830
|
+
- Timezone: ${blueprint.cron.timezone}
|
|
17831
|
+
- Jobs: ${jobs.map((job) => job.name).join(", ")}` : "- Disabled by input. Still create rank_tracker_runs for manual runs.",
|
|
17832
|
+
"",
|
|
17833
|
+
"## Implementation Prompt",
|
|
17834
|
+
"```text",
|
|
17835
|
+
implementationPrompt,
|
|
17836
|
+
"```"
|
|
17837
|
+
].join("\n");
|
|
17838
|
+
return {
|
|
17839
|
+
content: [{ type: "text", text }],
|
|
17840
|
+
structuredContent: blueprint
|
|
17841
|
+
};
|
|
17842
|
+
}
|
|
17843
|
+
var DEFAULT_MODES;
|
|
17844
|
+
var init_rank_tracker_blueprint = __esm({
|
|
17845
|
+
"src/mcp/rank-tracker-blueprint.ts"() {
|
|
17846
|
+
"use strict";
|
|
17847
|
+
DEFAULT_MODES = ["maps", "organic", "ai_overview", "paa"];
|
|
17848
|
+
}
|
|
17849
|
+
});
|
|
17850
|
+
|
|
17471
17851
|
// src/mcp/paa-mcp-server.ts
|
|
17472
17852
|
function liveWebToolAnnotations(title) {
|
|
17473
17853
|
return {
|
|
@@ -17478,6 +17858,15 @@ function liveWebToolAnnotations(title) {
|
|
|
17478
17858
|
openWorldHint: true
|
|
17479
17859
|
};
|
|
17480
17860
|
}
|
|
17861
|
+
function localPlanningToolAnnotations(title) {
|
|
17862
|
+
return {
|
|
17863
|
+
title,
|
|
17864
|
+
readOnlyHint: true,
|
|
17865
|
+
destructiveHint: false,
|
|
17866
|
+
idempotentHint: true,
|
|
17867
|
+
openWorldHint: false
|
|
17868
|
+
};
|
|
17869
|
+
}
|
|
17481
17870
|
function listSavedReports() {
|
|
17482
17871
|
try {
|
|
17483
17872
|
const dir = outputBaseDir();
|
|
@@ -17611,6 +18000,13 @@ function registerPaaExtractorMcpTools(server, executor, options = {}) {
|
|
|
17611
18000
|
outputSchema: DirectoryWorkflowOutputSchema,
|
|
17612
18001
|
annotations: liveWebToolAnnotations("Directory Workflow: Markets + Maps")
|
|
17613
18002
|
}, async (input) => formatDirectoryWorkflow(await executor.directoryWorkflow(input), input));
|
|
18003
|
+
server.registerTool("rank_tracker_blueprint", {
|
|
18004
|
+
title: "Rank Tracker Blueprint Builder",
|
|
18005
|
+
description: "Generate a build-ready database, cron/heartbeat, ingestion, metrics, and AI implementation prompt for a rank tracker powered by MCP Scraper. Supports Maps rankings through directory_workflow/maps_search, organic rankings through search_serp, AI Overview citation tracking, and People Also Ask source presence tracking. This tool is local planning only; it does not call the web or spend credits.",
|
|
18006
|
+
inputSchema: RankTrackerBlueprintInputSchema,
|
|
18007
|
+
outputSchema: RankTrackerBlueprintOutputSchema,
|
|
18008
|
+
annotations: localPlanningToolAnnotations("Rank Tracker Blueprint Builder")
|
|
18009
|
+
}, async (input) => buildRankTrackerBlueprint(input));
|
|
17614
18010
|
server.registerTool("credits_info", {
|
|
17615
18011
|
title: "MCP Scraper Credits & Costs",
|
|
17616
18012
|
description: "Answer questions about MCP Scraper credits: current credit balance, what a specific tool/action costs, the full cost table, and optionally recent credit ledger entries. Does not expose payment methods or credit card information.",
|
|
@@ -17635,6 +18031,7 @@ var init_paa_mcp_server = __esm({
|
|
|
17635
18031
|
init_version();
|
|
17636
18032
|
init_mcp_response_formatter();
|
|
17637
18033
|
init_mcp_tool_schemas();
|
|
18034
|
+
init_rank_tracker_blueprint();
|
|
17638
18035
|
init_mcp_response_formatter();
|
|
17639
18036
|
}
|
|
17640
18037
|
});
|