mcp-scraper 0.44.2 → 0.44.4

package/README.md

@@ -90,7 +90,7 @@ Build the branded one-click bundle:
 npm run build:mcpb
 ```
 
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.44.2`, SHA-256 `6ce5675c82d1fa0b7c645e0337aaac97de7760282067e5744259e8d0a4f03fc2`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
+The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.44.4`, SHA-256 `707dbcce1bafc04fc49f3a12cfc4ab43710e686e2f31102767ed120a3e8ff474`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
 
 The MCPB install exposes every tool — web-intelligence plus all `browser_*` tools — through the one `mcp-scraper` server.
 
@@ -233,7 +233,7 @@ The `mcp-scraper` server (and the MCPB bundle, which runs it) exposes both secti
 
 All MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The unified server exposes 215 tools: 114 scraper, browser, workflow, billing, and connected-service tools plus 101 durable-memory tools. The scraper inventory includes governed Local Sourcebook tools that follow a Memory-style contract/tag/prepare/validate/capture sequence before paid acquisition, plus Transparent Commons tools for public entity search, planning, validation, governed contribution, ledgers, needs-link discovery, and saved filters. Local Sourcebook publication remains administrator-only. Release verification compares the exact local and hosted tool-name sets, not only the count.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The unified server exposes 215 tools: 114 scraper, browser, workflow, billing, and connected-service tools plus 101 durable-memory tools. The scraper inventory includes governed Local Sourcebook tools that follow a Memory-style contract/tag/prepare/validate/capture sequence before paid acquisition, plus Transparent Commons tools for public entity search, planning, validation, governed contribution, ledgers, needs-link discovery, and saved filters. Successful evidence-compiled Local Sourcebook revisions publish automatically to their canonical `localsourcebook.com` category profile and review URLs; administrator controls handle exceptional rejection or unpublishing. Release verification compares the exact local and hosted tool-name sets, not only the count.
 
 For contract parity, stdio and MCPB memory calls invoke the matching public tool on the hosted MCP Scraper `/mcp` endpoint. The hosted aggregate runtime owns MCP Scraper-specific billing, scheduling, credential, and in-process cutover policy; its internal `/memory/mcp-call` bridge is a fallback to the standalone memory service, not the public stdio execution path. Direct `mcp-memory` OAuth and stdio clients continue to use `memory.mcpscraper.dev` and must be verified as a separate dependent release surface.
 

package/dist/bin/api-server.cjs

@@ -24789,6 +24789,14 @@ async function listPublicLocalSourcebook(category, state) {
   const result = await getDb().execute({ sql: `SELECT r.payload_json FROM local_sourcebook_submissions s JOIN local_sourcebook_revisions r ON r.submission_id = s.id AND r.revision = s.published_revision WHERE s.category = ? AND s.state = ? AND s.published_revision IS NOT NULL AND s.status NOT IN ('unpublished', 'rejected') ORDER BY s.business_name`, args: [category, state] });
   return result.rows.map((row) => parseJson2(row.payload_json));
 }
+async function listAllPublicLocalSourcebook(category) {
+  await ensureLocalSourcebookSchema();
+  const result = category ? await getDb().execute({
+    sql: `SELECT r.payload_json FROM local_sourcebook_submissions s JOIN local_sourcebook_revisions r ON r.submission_id = s.id AND r.revision = s.published_revision WHERE s.category = ? AND s.published_revision IS NOT NULL AND s.status NOT IN ('unpublished', 'rejected') ORDER BY s.state, s.business_name`,
+    args: [category]
+  }) : await getDb().execute(`SELECT r.payload_json FROM local_sourcebook_submissions s JOIN local_sourcebook_revisions r ON r.submission_id = s.id AND r.revision = s.published_revision WHERE s.published_revision IS NOT NULL AND s.status NOT IN ('unpublished', 'rejected') ORDER BY s.category, s.state, s.business_name`);
+  return result.rows.map((row) => parseJson2(row.payload_json));
+}
 async function getPublicLocalSourcebook(category, state, slug2) {
   await ensureLocalSourcebookSchema();
   const result = await getDb().execute({ sql: `SELECT r.payload_json FROM local_sourcebook_submissions s JOIN local_sourcebook_revisions r ON r.submission_id = s.id AND r.revision = s.published_revision WHERE s.category = ? AND s.state = ? AND s.slug = ? AND s.published_revision IS NOT NULL AND s.status NOT IN ('unpublished', 'rejected') LIMIT 1`, args: [category, state, slug2] });
@@ -25444,6 +25452,50 @@ var init_MapsExtractor = __esm({
   }
 });
 
+// src/api/local-sourcebook-public-urls.ts
+function rootDomain() {
+  const configured = process.env.LOCAL_SOURCEBOOK_PUBLIC_ROOT_DOMAIN?.trim() || DEFAULT_ROOT_DOMAIN;
+  return configured.replace(/^https?:\/\//i, "").replace(/^www\./i, "").replace(/\/.*$/, "").replace(/\.+$/, "").toLowerCase();
+}
+function localSourcebookPublicUrls(category, state, slug2) {
+  const categoryUrl = `https://${CATEGORY_PUBLIC_SLUGS[category]}.${rootDomain()}`;
+  const statePath = `/${state.toLowerCase()}`;
+  const profilePath = `${statePath}/${slug2}/`;
+  const reviewsPath = `${statePath}/${slug2}/reviews`;
+  return {
+    categoryUrl,
+    nationwideDirectoryUrl: `${categoryUrl}/directory`,
+    stateUrl: `${categoryUrl}${statePath}`,
+    profileUrl: `${categoryUrl}${profilePath}`,
+    reviewsUrl: `${categoryUrl}${reviewsPath}`,
+    profilePath,
+    reviewsPath
+  };
+}
+function localSourcebookPublicationLinks(submission) {
+  return {
+    ...localSourcebookPublicUrls(submission.category, submission.state, submission.slug),
+    publicationStatus: submission.status,
+    isLive: submission.status === "published" && submission.publishedRevision !== null
+  };
+}
+var DEFAULT_ROOT_DOMAIN, CATEGORY_PUBLIC_SLUGS;
+var init_local_sourcebook_public_urls = __esm({
+  "src/api/local-sourcebook-public-urls.ts"() {
+    "use strict";
+    DEFAULT_ROOT_DOMAIN = "localsourcebook.com";
+    CATEGORY_PUBLIC_SLUGS = {
+      home: "home-services",
+      professional: "professional",
+      restaurants: "restaurants",
+      financial: "financial",
+      realestate: "realestate",
+      auto: "auto",
+      wellness: "wellness"
+    };
+  }
+});
+
 // src/api/local-sourcebook-compiler.ts
 function digest(value) {
   return (0, import_node_crypto13.createHash)("sha256").update(value).digest("hex").slice(0, 20);
@@ -25719,6 +25771,7 @@ function compileLocalSourcebookListing(input) {
       evidenceIds: mapsEvidenceId ? [mapsEvidenceId] : []
     }
   ];
+  const publicUrls = localSourcebookPublicUrls(input.submission.category, input.submission.state, input.submission.slug);
   const listing = {
     id: input.submission.id,
     publicationRevision: input.submission.draftRevision + 1,
@@ -25727,6 +25780,14 @@ function compileLocalSourcebookListing(input) {
     state: input.submission.state.toLowerCase(),
     stateName: STATE_NAMES[input.submission.state.toUpperCase()] ?? input.submission.state.toUpperCase(),
     slug: input.submission.slug,
+    canonicalPaths: {
+      profile: publicUrls.profilePath,
+      reviews: publicUrls.reviewsPath
+    },
+    canonicalUrls: {
+      profile: publicUrls.profileUrl,
+      reviews: publicUrls.reviewsUrl
+    },
     name: maps?.name || input.submission.businessName,
     summary,
     description,
@@ -25790,6 +25851,7 @@ var init_local_sourcebook_compiler = __esm({
   "src/api/local-sourcebook-compiler.ts"() {
     "use strict";
     import_node_crypto13 = require("crypto");
+    init_local_sourcebook_public_urls();
     CATEGORY_LABELS = {
       home: "Home & property services",
       professional: "Professional services",
@@ -36621,7 +36683,11 @@ function getLocalSourcebookContract(category) {
     categories: LOCAL_SOURCEBOOK_CATEGORIES,
     selectedCategory: category ?? null,
     categoryTag: category ? CATEGORY_TAG[category] : null,
-    canonicalRoutes: { profile: "/{state}/{business-slug}/", reviews: "/{state}/{business-slug}/reviews" },
+    canonicalRoutes: {
+      nationwideDirectory: "https://{category}.localsourcebook.com/directory",
+      profile: "https://{category}.localsourcebook.com/{state}/{business-slug}/",
+      reviews: "https://{category}.localsourcebook.com/{state}/{business-slug}/reviews"
+    },
     requiredCreateFields: ["category", "state", "businessName", "websiteUrl", "tags", "idempotencyKey"],
     systemCompiledSections: ["identity", "services/products", "service areas/locations", "genuine media", "reviews and theme counts", "evidence", "answered FAQ", "collection receipts"],
     workflow: ["list-local-sourcebook-tags", "get-local-sourcebook-contract", "prepare-local-sourcebook-write", "validate-local-sourcebook-write", "local-sourcebook-capture", "local_sourcebook_submission_status"],
@@ -36641,9 +36707,10 @@ async function prepareLocalSourcebookWrite(input) {
     ...parsed.tags.filter((tag) => tag !== categoryTag).map((tag) => ({ tag }))
   ];
   const tagResolutions = await resolveLocalSourcebookTags(candidates);
+  const publicUrls = localSourcebookPublicUrls(parsed.category, parsed.state, slug2);
   return {
     ok: true,
-    route: { category: parsed.category, state: parsed.state, slug: slug2, profile: `/${parsed.state}/${slug2}/`, reviews: `/${parsed.state}/${slug2}/reviews` },
+    route: { category: parsed.category, state: parsed.state, slug: slug2, ...publicUrls },
     proposedWrite: { ...parsed, slug: slug2, tags: tagResolutions.filter((item) => item.action === "reuse").map((item) => item.tag) },
     contract: getLocalSourcebookContract(parsed.category),
     tagResolutions,
@@ -36709,6 +36776,7 @@ var init_local_sourcebook_governance = __esm({
     import_zod29 = require("zod");
     init_db();
     init_local_sourcebook_repository();
+    init_local_sourcebook_public_urls();
     DIRECTORY_TAG_SEEDS = [
       { tag: "home-services", description: "Businesses that maintain, repair, improve, or protect homes and property.", aliases: ["home service", "home and property services"] },
       { tag: "professional-services", description: "Local professional and business service providers.", aliases: ["professional service"] },
@@ -36759,6 +36827,18 @@ var init_local_sourcebook_governance = __esm({
 });
 
 // src/api/local-sourcebook-routes.ts
+function withCanonicalLocalSourcebookUrls(listing) {
+  const category = import_zod30.z.enum(LOCAL_SOURCEBOOK_CATEGORIES).safeParse(listing.category);
+  const state = typeof listing.state === "string" ? listing.state : "";
+  const listingSlug = typeof listing.slug === "string" ? listing.slug : "";
+  if (!category.success || !state || !listingSlug) return listing;
+  const publicUrls = localSourcebookPublicUrls(category.data, state, listingSlug);
+  return {
+    ...listing,
+    canonicalPaths: { profile: publicUrls.profilePath, reviews: publicUrls.reviewsPath },
+    canonicalUrls: { profile: publicUrls.profileUrl, reviews: publicUrls.reviewsUrl }
+  };
+}
 function acquisitionPlan(websiteUrl) {
   return {
     website: { startUrl: websiteUrl, mode: "broad_site_crawl", capture: ["homepage", "services", "products", "locations", "service_areas", "about", "team_staff", "contact", "policies"] },
@@ -36778,6 +36858,7 @@ var init_local_sourcebook_routes = __esm({
     init_local_sourcebook_dispatch();
     init_local_sourcebook_governance();
     init_local_sourcebook_repository();
+    init_local_sourcebook_public_urls();
     slug = (value) => value.toLowerCase().trim().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "").slice(0, 100);
     SubmitSchema = import_zod30.z.object({
       category: import_zod30.z.enum(LOCAL_SOURCEBOOK_CATEGORIES),
@@ -36826,6 +36907,7 @@ var init_local_sourcebook_routes = __esm({
           return c.json({ error: "idempotency_key_required", message: "A stable idempotencyKey is required for a new Local Sourcebook capture." }, 400);
         }
         const listingSlug = identity.slug ?? slug(identity.businessName);
+        const publicUrls = localSourcebookPublicUrls(identity.category, identity.state, listingSlug);
         const submission = await createLocalSourcebookSubmission({
           ownerUserId: Number(user.id),
           category: identity.category,
@@ -36842,7 +36924,8 @@ var init_local_sourcebook_routes = __esm({
             name: identity.businessName,
             websiteUrl: identity.websiteUrl,
             tags: validation.normalizedTags,
-            canonicalPaths: { profile: `/${identity.state}/${listingSlug}`, reviews: `/${identity.state}/${listingSlug}/reviews` },
+            canonicalPaths: { profile: publicUrls.profilePath, reviews: publicUrls.reviewsPath },
+            canonicalUrls: { profile: publicUrls.profileUrl, reviews: publicUrls.reviewsUrl },
             verificationState: "submitted",
             acquisitionPlan: acquisitionPlan(identity.websiteUrl),
             evidence: [],
@@ -36852,7 +36935,7 @@ var init_local_sourcebook_routes = __esm({
         });
         await persistLocalSourcebookTags(submission.id, Number(user.id), validation.normalizedTags, parsed.data.tagDecisions);
         const dispatched = await dispatchLocalSourcebookAcquisition(submission.id).catch(() => false);
-        return c.json({ ok: true, valid: true, submission, normalizedTags: validation.normalizedTags, acquisition: { queued: true, dispatched }, acquisitionPlan: acquisitionPlan(identity.websiteUrl) }, 202);
+        return c.json({ ok: true, valid: true, submission, publicUrls: localSourcebookPublicationLinks(submission), normalizedTags: validation.normalizedTags, acquisition: { queued: true, dispatched }, acquisitionPlan: acquisitionPlan(identity.websiteUrl) }, 202);
       } catch (error) {
         const message = error instanceof Error ? error.message : String(error);
         const conflict = message.includes("revision conflict") ? "revision_conflict" : message.includes("idempotency key conflicts") ? "idempotency_conflict" : null;
@@ -36866,6 +36949,7 @@ var init_local_sourcebook_routes = __esm({
       if (!parsed.success) return c.json({ error: "invalid_submission", issues: parsed.error.issues }, 400);
       const input = parsed.data;
       const listingSlug = input.slug ?? slug(input.businessName);
+      const publicUrls = localSourcebookPublicUrls(input.category, input.state, listingSlug);
       try {
         const submission = await createLocalSourcebookSubmission({
           ownerUserId: Number(user.id),
@@ -36878,7 +36962,8 @@ var init_local_sourcebook_routes = __esm({
             slug: listingSlug,
             name: input.businessName,
             websiteUrl: input.websiteUrl,
-            canonicalPaths: { profile: `/${input.state}/${listingSlug}`, reviews: `/${input.state}/${listingSlug}/reviews` },
+            canonicalPaths: { profile: publicUrls.profilePath, reviews: publicUrls.reviewsPath },
+            canonicalUrls: { profile: publicUrls.profileUrl, reviews: publicUrls.reviewsUrl },
             verificationState: "submitted",
             acquisitionPlan: acquisitionPlan(input.websiteUrl),
             evidence: [],
@@ -36887,7 +36972,7 @@ var init_local_sourcebook_routes = __esm({
           }
         });
         const dispatched = await dispatchLocalSourcebookAcquisition(submission.id).catch(() => false);
-        return c.json({ ok: true, submission, acquisition: { queued: true, dispatched }, acquisitionPlan: acquisitionPlan(input.websiteUrl) }, 202);
+        return c.json({ ok: true, submission, publicUrls: localSourcebookPublicationLinks(submission), acquisition: { queued: true, dispatched }, acquisitionPlan: acquisitionPlan(input.websiteUrl) }, 202);
       } catch (error) {
         return c.json({ error: "submission_conflict", message: error instanceof Error ? error.message : String(error) }, 409);
       }
@@ -36896,7 +36981,7 @@ var init_local_sourcebook_routes = __esm({
       const userId = Number(c.get("user").id);
       const submission = await getLocalSourcebookSubmission(c.req.param("id"), userId);
       if (!submission) return c.json({ error: "not_found" }, 404);
-      return c.json({ ok: true, submission, draft: await getLocalSourcebookDraft(submission.id, userId) });
+      return c.json({ ok: true, submission, publicUrls: localSourcebookPublicationLinks(submission), draft: await getLocalSourcebookDraft(submission.id, userId) });
     });
     localSourcebookApp.on(["PATCH", "POST"], "/submissions/:id/draft", createApiKeyAuth(), (c) => c.json({
       error: "evidence_refresh_required",
@@ -36909,16 +36994,18 @@ var init_local_sourcebook_routes = __esm({
         const submission = await queueLocalSourcebookRefresh(c.req.param("id"), Number(user.id));
         if (!submission) return c.json({ error: "not_found" }, 404);
         const dispatched = await dispatchLocalSourcebookAcquisition(submission.id).catch(() => false);
-        return c.json({ ok: true, submission, acquisition: { queued: true, dispatched }, acquisitionPlan: acquisitionPlan(submission.websiteUrl) }, 202);
+        return c.json({ ok: true, submission, publicUrls: localSourcebookPublicationLinks(submission), acquisition: { queued: true, dispatched }, acquisitionPlan: acquisitionPlan(submission.websiteUrl) }, 202);
       } catch (error) {
         return c.json({ error: "moderated_listing", message: error instanceof Error ? error.message : String(error) }, 409);
       }
     });
     publicLocalSourcebookApp = new import_hono17.Hono();
-    publicLocalSourcebookApp.get("/:category/:state", async (c) => c.json({ ok: true, listings: await listPublicLocalSourcebook(c.req.param("category"), c.req.param("state").toLowerCase()) }));
+    publicLocalSourcebookApp.get("/", async (c) => c.json({ ok: true, listings: (await listAllPublicLocalSourcebook()).map(withCanonicalLocalSourcebookUrls) }));
+    publicLocalSourcebookApp.get("/:category", async (c) => c.json({ ok: true, listings: (await listAllPublicLocalSourcebook(c.req.param("category"))).map(withCanonicalLocalSourcebookUrls) }));
+    publicLocalSourcebookApp.get("/:category/:state", async (c) => c.json({ ok: true, listings: (await listPublicLocalSourcebook(c.req.param("category"), c.req.param("state").toLowerCase())).map(withCanonicalLocalSourcebookUrls) }));
     publicLocalSourcebookApp.get("/:category/:state/:slug", async (c) => {
       const listing = await getPublicLocalSourcebook(c.req.param("category"), c.req.param("state").toLowerCase(), c.req.param("slug"));
-      return listing ? c.json({ ok: true, listing }) : c.json({ error: "not_found" }, 404);
+      return listing ? c.json({ ok: true, listing: withCanonicalLocalSourcebookUrls(listing) }) : c.json({ error: "not_found" }, 404);
     });
     adminLocalSourcebookApp = new import_hono17.Hono();
     adminLocalSourcebookApp.use("*", adminAuth);
@@ -40932,7 +41019,7 @@ var PACKAGE_VERSION;
 var init_version = __esm({
   "src/version.ts"() {
     "use strict";
-    PACKAGE_VERSION = "0.44.2";
+    PACKAGE_VERSION = "0.44.4";
   }
 });
 
@@ -41103,6 +41190,8 @@ Multi-step orchestrations \u2014 prefer these over hand-chaining primitives when
   publishes automatically. Never supply owner-authored services, reviews, images, or other public claims.
 - Read progress and the compiled record with **local_sourcebook_submission_status**. Use
   **local_sourcebook_refresh** only when the owner intentionally wants to pay for a new acquisition pass.
+- Capture and status responses return \`publicUrls.profileUrl\` and \`publicUrls.reviewsUrl\`. Always give those
+  exact links to the user; do not report a listing as live without including its clickable profile URL.
 - The last published revision remains public during refresh. Administrators can reject or unpublish an
   exceptional record, but routine subscriber publication does not require administrator review.
 
@@ -45182,7 +45271,7 @@ function registerPaaExtractorMcpTools(server, executor, options = {}) {
   }, async (input) => formatDirectoryWorkflow(await executor.directoryWorkflowStatus(input), input, ctx));
   server.registerTool("get-local-sourcebook-contract", {
     title: "Get Local Sourcebook Contract",
-    description: "Read the governed listing purpose, supported categories, canonical profile/review routes, required sections, acquisition limits, tag policy, ownership boundary, and admin-only publication rule. Call this before composing a listing when its required shape is uncertain.",
+    description: "Read the governed listing purpose, supported categories, exact LocalSourcebook.com profile/review URLs, nationwide directory URL, required sections, acquisition limits, tag policy, ownership boundary, and automatic evidence-publication rule. Call this before composing a listing when its required shape is uncertain.",
     inputSchema: GetLocalSourcebookContractInputSchema,
     outputSchema: recordOutputSchema("get-local-sourcebook-contract", LocalSourcebookOutputSchema),
     annotations: localPlanningToolAnnotations("Get Local Sourcebook Contract")
@@ -45217,14 +45306,14 @@ function registerPaaExtractorMcpTools(server, executor, options = {}) {
   }, async (input) => executor.validateLocalSourcebookWrite(input));
   server.registerTool("local-sourcebook-capture", {
     title: "Capture Governed Local Sourcebook Listing",
-    description: "Strict owner-scoped write path after list, contract, prepare, and validate. Capture registers canonical tags and queues paid website, exact-place, review, service-area, staff, and genuine-media acquisition. A successful system-compiled evidence revision publishes automatically; owner-authored public claims are not accepted.",
+    description: "Strict owner-scoped write path after list, contract, prepare, and validate. Capture registers canonical tags, returns the exact LocalSourcebook.com profile and reviews URLs, and queues paid website, exact-place, review, service-area, staff, and genuine-media acquisition. A successful system-compiled evidence revision publishes automatically; owner-authored public claims are not accepted.",
     inputSchema: LocalSourcebookCaptureInputSchema,
     outputSchema: recordOutputSchema("local-sourcebook-capture", LocalSourcebookOutputSchema),
     annotations: liveWebToolAnnotations("Capture Governed Local Sourcebook Listing")
   }, async (input) => executor.localSourcebookCapture(input));
   server.registerTool("local_sourcebook_submission_status", {
     title: "Local Sourcebook Submission Status",
-    description: "Read the authenticated caller\u2019s listing draft, enrichment coverage, immutable revision number, and publication state.",
+    description: "Read the authenticated caller\u2019s listing draft, enrichment coverage, immutable revision number, publication state, and exact live LocalSourcebook.com profile and reviews URLs.",
     inputSchema: LocalSourcebookSubmissionStatusInputSchema,
     outputSchema: recordOutputSchema("local_sourcebook_submission_status", LocalSourcebookOutputSchema),
     annotations: localPlanningToolAnnotations("Local Sourcebook Submission Status")