mcp-scraper 0.79.4 → 0.81.0

package/CHANGELOG.md

@@ -4,6 +4,50 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ## [Unreleased]
 
+## [0.81.0] - 2026-08-31
+
+### Added
+
+- Added a Kernel-only Reddit workflow that searches DuckDuckGo with a `site:reddit.com` query, switches the same browser to a residential proxy before clicking the selected result, and reads modern Reddit posts plus bounded rendered-comment expansion through dedicated search, thread, and combined REST endpoints.
+- Added a bounded managed-browser backup for Reddit thread hydration after the primary Kernel attempt fails or returns fewer than the semantic target, capped at 25 comments with measured bandwidth, duration, CAPTCHA, closure, and provider-cost telemetry.
+
+### Changed
+
+- Routed the production `reddit_thread` and `reddit_trending` MCP tools through modern Reddit on Kernel residential sessions, with DuckDuckGo site search for trend discovery; removed Google and old Reddit from their active execution path while preserving tool names, billing rates, bounded partial results, and refunds.
+- Cost probes now include Reddit Kernel sessions and any managed-browser fallback bytes and cost in the same request receipt, and identify when the backup contributed to total cost.
+
+### Fixed
+
+- Reconciled delayed managed-browser fallback telemetry back into its linked cost probe, correcting stored vendor totals and both margin estimates after provider finalization; cost probes now label unresolved provider cost as pending instead of reporting a false zero.
+- Measured the package-entrypoint size guard against files selected by the package publish allowlist while continuing to reject source maps anywhere in the built distribution.
+- Account publishing lists accept hosted routing metadata while retaining strict pagination, filtering, and account isolation.
+- Added a private, paginated publishing inventory for each account’s wiki contributions, publication subdomains and editions, and Local Sourcebook listings, with canonical links and current publication status.
+- Validated Twilio and HubSpot X-Ray webhook signatures against the exact public callback URL instead of an internal deployment origin or Vercel rewrite query, allowing genuine provider events to become live evidence after reverse-proxy delivery.
+
+## [0.80.0] - 2026-08-29
+
+### Added
+
+- Added a resumable Personal Assistant setup journey that separates product availability, operator activation, provider connections, sender registration and binding, personal-phone enrollment and consent, first-message testing, and per-feature readiness without erasing healthy state when one dependency is unavailable.
+- Added verified recipient enrollment through inbound `START` or a short-lived web challenge, with masked owner-scoped contact records, versioned consent disclosures, safe resend and removal flows, and explicit sender and destination identity throughout Assistant and Scheduling.
+- Added owner-safe diagnostics, fixed-message delivery tests, encrypted unknown-sender quarantine and authenticated claim flows, MMS and attachment lifecycle states, cost and usage controls, and reviewed export, revocation, removal, deletion, and number-release ceremonies.
+- Added `assistant_diagnostics_get` and `assistant_delivery_test`, bringing the public catalog to 356 tools and the governed Assistant surface to 17 tools while preserving its two resources and one setup prompt.
+
+### Changed
+
+- Made Scheduling the only Personal Assistant automation lifecycle owner, with exact draft, readiness, reviewed persistence, activation, accepted readback, connection operations, browser login grants, delivery destinations, fallbacks, quiet hours, urgent exceptions, retry, expiry, and failure-notification parity.
+- Expanded the authenticated Assistant owner API to 77 closed-schema operations and regenerated the OpenAPI, MCP manifest, CLI, cURL, Node, and Python contracts without generic or unknown schema fallbacks.
+- Replaced generic hosted-adapter and “finish setup” failures with component-owned activation blockers, truthful connection health, safe retry guidance, and opaque support receipts.
+
+### Security
+
+- Kept regulatory contact information separate from delivery consent, prevented ambiguous tenant routing and hidden quarantined content from reaching an assistant, and required exact owner, sender, recipient, consent, grant, schedule, and idempotency evidence at every external-write boundary.
+- Versioned `START`, `STOP`, `HELP`, privacy, terms, enrollment, first-message, and carrier-submission copy so live messaging policy can evolve without rewriting submitted provider evidence.
+
+### Migration
+
+- Advance the Personal Assistant database schema and deployment contract to `2026-08-28.13` before enabling the new recipient, quarantine, first-text diagnostic, or lifecycle routes. Hosted outbound messaging remains unavailable until the configured vault, Twilio resources, provider clients, administrator write authority, and carrier registration gates are verified.
+
 ## [0.79.4] - 2026-08-29
 
 ### Fixed
@@ -1500,7 +1544,11 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 - Write actions remain unavailable until the account owner explicitly enables them.
 - Provider-specific connection data is normalized into one agent-facing contract.
 
-[Unreleased]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.79.2...HEAD
+[Unreleased]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.81.0...HEAD
+[0.81.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.80.0...v0.81.0
+[0.80.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.79.4...v0.80.0
+[0.79.4]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.79.3...v0.79.4
+[0.79.3]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.79.2...v0.79.3
 [0.79.2]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.79.1...v0.79.2
 [0.79.1]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.79.0...v0.79.1
 [0.79.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.78.0...v0.79.0

package/README.md

@@ -2,6 +2,8 @@
 
 MCP Scraper is an MCP server for live web intelligence tools backed by `https://mcpscraper.dev`.
 
+The authenticated [Kernel Reddit API](docs/kernel-reddit.md) provides an ordered DuckDuckGo search, runtime residential-proxy handoff, real result click, and modern Reddit reader at `/kernel-reddit`.
+
 ## Product repository
 
 This repository owns the complete private product runtime while preserving independent service and
@@ -173,7 +175,7 @@ Build the branded one-click bundle:
 npm run build:mcpb
 ```
 
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.79.4`, SHA-256 `8c7db6b3188db21d9328ff8bf408d72bd5cd911f93afaf04c7fb4d68f592ca85`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest.
+The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.81.0`, SHA-256 `866faf8d18568514b29f407c6789bc779e895c9780cfcee27eb958cb079cf244`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest.
 
 The MCPB install exposes every tool — web-intelligence plus all `browser_*` tools — through the one `mcp-scraper` server.
 
@@ -335,7 +337,7 @@ The `mcp-scraper` server (and the MCPB bundle, which runs it) exposes both secti
 
 All MCP tools return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step, plus readable text content for compatibility. Runtime `tools/list` omits output schemas so strict clients can register the complete catalog; the generated developer manifest retains every canonical output schema for validation and typed SDK generation. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The unified server exposes 354 tools: 253 scraper, browser, workflow, billing, connected-service, and personal-assistant tools plus 101 durable-memory tools. The scraper-side inventory includes complete Gmail selection, message, attachment, export, bulk-action, and Memory-import workflows; governed personal-assistant commands, messaging, approvals, grants, number setup, and execution readback; durable PAA starts and status; rendered site-content similarity; governed Local Sourcebook and Transparent Commons workflows; direct site-export reads; Editorial Reading Room and News Publisher templates; and production X-Ray analytics, attribution, CRM policy and receipt, campaign, export, and scheduled-report tools. Successful evidence-compiled Local Sourcebook revisions publish automatically to their canonical `localsourcebook.com` category profile and review URLs; administrator controls handle exceptional rejection or unpublishing. Release verification compares the exact local and hosted tool-name sets, not only the count.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The unified server exposes 356 tools: 255 scraper, browser, workflow, billing, connected-service, and personal-assistant tools plus 101 durable-memory tools. The scraper-side inventory includes complete Gmail selection, message, attachment, export, bulk-action, and Memory-import workflows; governed personal-assistant commands, messaging, approvals, grants, number setup, and execution readback; durable PAA starts and status; rendered site-content similarity; governed Local Sourcebook and Transparent Commons workflows; direct site-export reads; Editorial Reading Room and News Publisher templates; and production X-Ray analytics, attribution, CRM policy and receipt, campaign, export, and scheduled-report tools. Successful evidence-compiled Local Sourcebook revisions publish automatically to their canonical `localsourcebook.com` category profile and review URLs; administrator controls handle exceptional rejection or unpublishing. Release verification compares the exact local and hosted tool-name sets, not only the count.
 
 For contract parity, stdio and MCPB memory calls invoke the matching public tool on the hosted MCP Scraper `/mcp` endpoint. The hosted aggregate runtime owns MCP Scraper-specific billing, scheduling, credential, and in-process cutover policy; its internal `/memory/mcp-call` bridge is a fallback to the standalone memory service, not the public stdio execution path. Direct `mcp-memory` OAuth and stdio clients continue to use `memory.mcpscraper.dev` and must be verified as a separate dependent release surface.
 

package/dist/bin/api-server.js

@@ -17,8 +17,8 @@ loadDotEnv();
 async function main() {
   const [{ serve }, { app, personalAssistantProductionStartup }, { startWorker }, { migrate }] = await Promise.all([
     import("@hono/node-server"),
-    import("../server-BMQKWVCZ.js"),
-    import("../worker-3LUFQWWJ.js"),
+    import("../server-QRYEMF4H.js"),
+    import("../worker-2MY77WYY.js"),
     import("../db-Z34LPZNR.js")
   ]);
   const PORT = parseInt(process.env.PORT ?? "3001");

package/dist/bin/mcp-scraper-cli.js

@@ -13,7 +13,7 @@ import "../chunk-VXLU74YZ.js";
 import "../chunk-GGZEC22A.js";
 import {
   PACKAGE_VERSION
-} from "../chunk-N63ZFFU2.js";
+} from "../chunk-RSXDHQBA.js";
 
 // src/cli/human-cli.ts
 import { Command } from "commander";

package/dist/bin/mcp-scraper-core.js

@@ -1,19 +1,19 @@
 #!/usr/bin/env node
 import {
   runMcpScraperStdio
-} from "../chunk-PQVLX5WT.js";
-import "../chunk-RUBI3OXR.js";
+} from "../chunk-Z63DQAO7.js";
+import "../chunk-XWXCLKXF.js";
 import "../chunk-PGJQDMC2.js";
 import "../chunk-DNM65UCK.js";
 import "../chunk-T3MZISOF.js";
 import "../chunk-6HAV7LCE.js";
-import "../chunk-VFCUGTCF.js";
+import "../chunk-PMONKQGJ.js";
 import "../chunk-OM7HVEJ3.js";
-import "../chunk-5UAFBKJR.js";
-import "../chunk-ETQ7J2EB.js";
+import "../chunk-OPQIGAFB.js";
+import "../chunk-M2A7YSJV.js";
 import "../chunk-VXLU74YZ.js";
 import "../chunk-GGZEC22A.js";
-import "../chunk-N63ZFFU2.js";
+import "../chunk-RSXDHQBA.js";
 import "../chunk-YXNDOQXN.js";
 
 // src/mcp/thorbit-restricted-tool-surface.ts

package/dist/bin/mcp-scraper-install.js

@@ -1,10 +1,10 @@
 #!/usr/bin/env node
 import {
   renderInstallTerminal
-} from "../chunk-ETQ7J2EB.js";
+} from "../chunk-M2A7YSJV.js";
 import {
   PACKAGE_VERSION
-} from "../chunk-N63ZFFU2.js";
+} from "../chunk-RSXDHQBA.js";
 
 // bin/mcp-scraper-install.ts
 var noColor = process.argv.includes("--no-color") || process.env.NO_COLOR !== void 0 || process.env.FORCE_COLOR === "0" || !process.stdout.isTTY;

package/dist/bin/mcp-stdio-server.js

@@ -1,19 +1,19 @@
 #!/usr/bin/env node
 import {
   runMcpScraperStdio
-} from "../chunk-PQVLX5WT.js";
-import "../chunk-RUBI3OXR.js";
+} from "../chunk-Z63DQAO7.js";
+import "../chunk-XWXCLKXF.js";
 import "../chunk-PGJQDMC2.js";
 import "../chunk-DNM65UCK.js";
 import "../chunk-T3MZISOF.js";
 import "../chunk-6HAV7LCE.js";
-import "../chunk-VFCUGTCF.js";
+import "../chunk-PMONKQGJ.js";
 import "../chunk-OM7HVEJ3.js";
-import "../chunk-5UAFBKJR.js";
-import "../chunk-ETQ7J2EB.js";
+import "../chunk-OPQIGAFB.js";
+import "../chunk-M2A7YSJV.js";
 import "../chunk-VXLU74YZ.js";
 import "../chunk-GGZEC22A.js";
-import "../chunk-N63ZFFU2.js";
+import "../chunk-RSXDHQBA.js";
 import "../chunk-YXNDOQXN.js";
 
 // bin/mcp-stdio-server.ts

package/dist/bin/paa-harvest.js

@@ -1,11 +1,11 @@
 #!/usr/bin/env node
 import {
   harvest
-} from "../chunk-KNGTQD6U.js";
+} from "../chunk-NH4QAH3X.js";
 import {
   browserServiceApiKey
 } from "../chunk-OM7HVEJ3.js";
-import "../chunk-5UAFBKJR.js";
+import "../chunk-OPQIGAFB.js";
 import "../chunk-OWF2JJKN.js";
 import "../chunk-YXNDOQXN.js";
 

package/dist/{chunk-ETQ7J2EB.js → chunk-M2A7YSJV.js}

@@ -1,6 +1,6 @@
 // release-message.json
 var release_message_default = {
-  message: "PAA harvests now expose exact interaction timing, distinguish interrupted work from proven frontier exhaustion, and install with a pinned schema runtime so the hosted and local 354-tool contracts remain identical."
+  message: "Reddit thread and trending tools now use residential browser retrieval with a bounded backup, honest partial-result handling, and measured fallback cost telemetry."
 };
 
 // src/install-terminal.ts
@@ -55,7 +55,7 @@ function renderInstallTerminal(options) {
     "1/1 install surfaces ready",
     colorize(`Newest in v${options.version}: ${release_message_default.message}`, "lime", color),
     "",
-    `${colorize("Tools", "cyan", color)}  ${colorize("(354 MCP tools)", "muted", color)}`,
+    `${colorize("Tools", "cyan", color)}  ${colorize("(356 MCP tools)", "muted", color)}`,
     toolRow("search", ["harvest_paa", "search_serp", "maps_search", "maps_place_intel"], color),
     toolRow("extract", ["extract_url", "map_site_urls", "extract_site", "audit_site", "directory_workflow"], color),
     toolRow("build", ["create_editorial_reading_room", "rank_tracker_workflow", "portable HTML"], color),

package/dist/{chunk-KNGTQD6U.js → chunk-NH4QAH3X.js}

@@ -8,7 +8,7 @@ import {
   currentCostContext,
   recordKernelSession,
   runWithCostContext
-} from "./chunk-5UAFBKJR.js";
+} from "./chunk-OPQIGAFB.js";
 import {
   DEFAULT_PROXY_MODE,
   HarvestOptionsSchema,
@@ -5120,7 +5120,9 @@ export {
   serpArrivalUrl,
   buildYouTubeChannelVideosUrl,
   BrowserDriver,
+  sanitizeProviderLocalMessage,
   validateProviderSessionId,
+  brightDataBrowserEndpoint,
   brightDataSerpEnabled,
   questionIdFor,
   classifyGoogleOutboundLink,

package/dist/{chunk-5UAFBKJR.js → chunk-OPQIGAFB.js}

@@ -33,7 +33,7 @@ var NANGO_USD_PER_CONNECTION_MONTH = envRate("NANGO_USD_PER_CONNECTION_MONTH", 1
 var NANGO_USD_PER_FUNCTION_RUN = envRate("NANGO_USD_PER_FUNCTION_RUN", 1e-4);
 var NANGO_USD_PER_PROXY_REQUEST = envRate("NANGO_USD_PER_PROXY_REQUEST", 1e-4);
 var NANGO_USD_PER_COMPUTE_SEC = envRate("NANGO_USD_PER_COMPUTE_SEC", 2e-4);
-var BRIGHTDATA_BROWSER_USD_PER_GB = envRate("BRIGHTDATA_BROWSER_USD_PER_GB", 0);
+var BRIGHTDATA_BROWSER_USD_PER_GB = envRate("BRIGHTDATA_BROWSER_USD_PER_GB", 8);
 function kernelCostUsd(ms, headful) {
   const sec = Math.max(0, ms) / 1e3;
   return sec * (headful ? KERNEL_HEADFUL_USD_PER_SEC : KERNEL_HEADLESS_USD_PER_SEC);
@@ -74,7 +74,7 @@ async function runCostTelemetryMigration() {
     SELECT
       (SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name IN ('kernel_session_log', 'vendor_usage_log', 'cost_probe_runs')) = 3
       AND (SELECT COUNT(*) FROM pragma_table_info('kernel_session_log') WHERE name IN ('proxy_source', 'proxy_type', 'method')) = 3
-      AND (SELECT COUNT(*) FROM pragma_table_info('vendor_usage_log') WHERE name IN ('method', 'source_key')) = 2
+      AND (SELECT COUNT(*) FROM pragma_table_info('vendor_usage_log') WHERE name IN ('method', 'source_key', 'provider_duration_ms', 'provider_captcha', 'provider_status')) = 5
       AND (SELECT COUNT(*) FROM sqlite_master WHERE type = 'index' AND name = 'vendor_usage_log_vendor_source_key') = 1
       AND (SELECT COUNT(*) FROM pragma_table_info('cost_probe_runs') WHERE name IN ('units', 'unit_type', 'mode')) = 3
       AS ready
@@ -132,6 +132,9 @@ async function runCostTelemetryMigration() {
       est_cost_usd REAL NOT NULL DEFAULT 0,
       error        TEXT,
       source_key   TEXT,
+      provider_duration_ms INTEGER,
+      provider_captcha INTEGER,
+      provider_status TEXT,
       created_at   TEXT NOT NULL DEFAULT (datetime('now'))
     )
   `);
@@ -146,6 +149,18 @@ async function runCostTelemetryMigration() {
     await db.execute(`ALTER TABLE vendor_usage_log ADD COLUMN source_key TEXT`);
   } catch {
   }
+  try {
+    await db.execute(`ALTER TABLE vendor_usage_log ADD COLUMN provider_duration_ms INTEGER`);
+  } catch {
+  }
+  try {
+    await db.execute(`ALTER TABLE vendor_usage_log ADD COLUMN provider_captcha INTEGER`);
+  } catch {
+  }
+  try {
+    await db.execute(`ALTER TABLE vendor_usage_log ADD COLUMN provider_status TEXT`);
+  } catch {
+  }
   await db.execute(`CREATE UNIQUE INDEX IF NOT EXISTS vendor_usage_log_vendor_source_key ON vendor_usage_log(vendor, source_key) WHERE source_key IS NOT NULL`);
   await db.execute(`
     CREATE TABLE IF NOT EXISTS cost_probe_runs (
@@ -228,8 +243,8 @@ async function recordVendorUsage(r) {
     const db = getDb();
     const result = await db.execute({
       sql: `INSERT OR IGNORE INTO vendor_usage_log
-        (id, op, probe_run_id, user_id, vendor, model, units, unit_type, est_cost_usd, error, method, source_key)
-        VALUES (?,?,?,?,?,?,?,?,?,?,?,?)`,
+        (id, op, probe_run_id, user_id, vendor, model, units, unit_type, est_cost_usd, error, method, source_key, provider_duration_ms, provider_captcha, provider_status)
+        VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
       args: [
         randomUUID(),
         r.op ?? ctx?.op ?? null,
@@ -242,7 +257,10 @@ async function recordVendorUsage(r) {
         vendorCostUsd(r.vendor, r.units),
         r.error ?? null,
         r.method ?? ctx?.subOp ?? null,
-        r.sourceKey ?? null
+        r.sourceKey ?? null,
+        r.providerDurationMs ?? null,
+        boolToInt(r.providerCaptcha),
+        r.providerStatus ?? null
       ]
     });
     return result.rowsAffected === 1;
@@ -262,6 +280,7 @@ export {
   currentCostContext,
   HEADLESS_OPS,
   vendorCostUsd,
+  migrateCostTelemetry,
   recordKernelSession,
   recordVendorUsage
 };

package/dist/{chunk-VFCUGTCF.js → chunk-PMONKQGJ.js}

@@ -213,8 +213,8 @@ var PUBLIC_PRODUCT_CONTRACT = {
     "freeSignupCredits": 0
   },
   "inventory": {
-    "totalTools": 354,
-    "scraperTools": 253,
+    "totalTools": 356,
+    "scraperTools": 255,
     "memoryTools": 101
   },
   "concurrencyPack": {

package/dist/{chunk-N63ZFFU2.js → chunk-RSXDHQBA.js}

@@ -1,5 +1,5 @@
 // src/version.ts
-var PACKAGE_VERSION = "0.79.4";
+var PACKAGE_VERSION = "0.81.0";
 
 export {
   PACKAGE_VERSION

package/dist/{chunk-3423R23T.js → chunk-VWC7XATD.js}

@@ -19,11 +19,11 @@ import {
   LedgerOperation,
   MC_PER_CREDIT,
   calculateConnectedUsageCharge
-} from "./chunk-VFCUGTCF.js";
+} from "./chunk-PMONKQGJ.js";
 import {
   recordVendorUsage,
   runWithCostContext
-} from "./chunk-5UAFBKJR.js";
+} from "./chunk-OPQIGAFB.js";
 import {
   debitMcIdempotent,
   getDb,

package/dist/{chunk-YBR32LOD.js → chunk-XBEUZJHS.js}

@@ -7,7 +7,7 @@ import {
 } from "./chunk-DNM65UCK.js";
 import {
   LedgerOperation
-} from "./chunk-VFCUGTCF.js";
+} from "./chunk-PMONKQGJ.js";
 import {
   CREDIT_LOT_TTL,
   getDb,