mcp-scraper 0.88.2 → 0.89.1

package/CHANGELOG.md

@@ -4,6 +4,42 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ## [Unreleased]
 
+## [0.89.1] - 2026-09-08
+
+### Changed
+
+- Added an operator-controlled path to isolate OAuth state in a dedicated Turso database with authority epochs, bounded cleanup, verified migration and rollback, and fail-closed maintenance behavior. Existing OAuth issuer, client setup, API-key authentication, and MCP tool contracts remain unchanged.
+- Bounded the root cron by admitting work lazily with explicit concurrency and deadline limits, so timed-out responses no longer eagerly start every maintenance task.
+
+### Fixed
+
+- Made authorization-code exchange and refresh-token rotation atomic and safely retryable after an uncertain database response, including deterministic replay of the one committed result under concurrent requests.
+- Accepted standards-compatible HTTPS, loopback, and reverse-domain native redirects while enforcing PKCE syntax, redirect matching, client binding, scope reduction, resource binding, and bounded dynamic registration.
+
+### Security
+
+- Replaced raw OAuth code and refresh-token storage in the canonical store with keyed digests and short-lived authenticated-encryption replay receipts, added key-retention checks, and kept destructive legacy-row cleanup behind a separate explicit gate.
+
+## [0.89.0] - 2026-09-04
+
+### Added
+
+- Collect Facebook profile and Page Reel URLs with `facebook_reels_inventory`, including partial results, stopping reasons, and deduplication against previously returned URLs. Anonymous scans return up to 60 URLs; a scan costs 4 credits and is refunded when it adds no new URLs.
+
+### Changed
+
+- Use neutral browser labels in Facebook extraction diagnostics and package the existing local API, library, and CLI with provider-name obfuscation.
+
+### Fixed
+
+- Dismiss optional Facebook login dialogs before extracting public videos and Reel inventories, and retry failed acquisition through a backup browser while preserving useful partial results.
+
+## [0.88.3] - 2026-09-02
+
+### Fixed
+
+- Give each `reddit_thread` retrieval a 300-second end-to-end deadline, with two 60-second Kernel attempts and two 60-second managed-browser backup attempts, instead of exhausting the full retry ladder in about 50 seconds. The MCP client now waits long enough to receive the endpoint's structured terminal result.
+
 ## [0.88.2] - 2026-09-02
 
 ### Fixed
@@ -35,7 +71,6 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 - Added optional `pages: 2` support to synchronous and durable PAA harvests, capturing a second organic result page before expanding questions on the untouched first page.
 - Added requested/captured page counts and bounded second-page outcomes to live results, durable recovery, the dashboard, and generated client contracts. A failed optional page-two capture preserves first-page evidence.
-
 ## [0.86.5] - 2026-09-02
 
 ### Fixed
@@ -1737,7 +1772,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 - Write actions remain unavailable until the account owner explicitly enables them.
 - Provider-specific connection data is normalized into one agent-facing contract.
 
-[Unreleased]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.88.0...HEAD
+[Unreleased]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.89.0...HEAD
 [0.88.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.87.0...v0.88.0
 [0.87.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.86.5...v0.87.0
 [0.86.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.85.0...v0.86.0
@@ -1880,3 +1915,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 [0.21.5]: https://github.com/VilovietaSEO/mcp-scraper/releases/tag/v0.21.5
 [0.21.4]: https://github.com/VilovietaSEO/mcp-scraper/releases/tag/v0.21.4
 [0.11.0]: https://github.com/VilovietaSEO/mcp-scraper/releases/tag/v0.11.0
+
+[0.89.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.88.3...v0.89.0
+[Unreleased]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.89.1...HEAD
+[0.89.1]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.89.0...v0.89.1

package/README.md

@@ -2,7 +2,7 @@
 
 MCP Scraper is an MCP server for live web intelligence tools backed by `https://mcpscraper.dev`.
 
-The authenticated [Kernel Reddit API](docs/kernel-reddit.md) provides an ordered DuckDuckGo search, runtime residential-proxy handoff, real result click, and modern Reddit reader at `/kernel-reddit`.
+The authenticated Reddit API supports search, result selection, and extraction from current Reddit pages.
 
 ## Product repository
 
@@ -175,7 +175,7 @@ Build the branded one-click bundle:
 npm run build:mcpb
 ```
 
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.88.2`, SHA-256 `353608a80e9c4b9879e3fa293aaa095a7f5c75b4b713f127d044775b329e455f`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest.
+The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.89.1`, SHA-256 `f7ea38f87f0bb1bd73900f3817b47f923eb98c18428ca1b6f010d95f489aac74`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest.
 
 The MCPB install exposes every tool — web-intelligence plus all `browser_*` tools — through the one `mcp-scraper` server.
 
@@ -259,6 +259,7 @@ Check `pagination.requestedPages`, `pagination.capturedPages`, and `pagination.p
 - `archive_read` — list files in a bounded public ZIP, read one selected UTF-8 text file in byte windows, or preserve the complete selected file in the tenant Library vault. Use it with the `bundleUrl` returned by `check_site_export`.
 - `youtube_harvest`
 - `youtube_transcribe`
+- `facebook_reels_inventory` — collect up to the requested number of public Reel URLs from a Facebook profile or Page, then pass selected URLs to `facebook_video_transcribe`. Anonymous scans return at most 60 URLs; access walls and time limits can return fewer. Partial results include a stopping reason and can be retained with `resumeUrls`. A scan costs 4 credits and is refunded when it adds no new URLs.
 - `facebook_ad_search`
 - `facebook_page_intel`
 - `facebook_ad_transcribe` — transcribe a direct Facebook ad video URL returned by `facebook_page_intel`.
@@ -350,7 +351,7 @@ The `mcp-scraper` server (and the MCPB bundle, which runs it) exposes both secti
 
 All MCP tools return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step, plus readable text content for compatibility. Runtime `tools/list` omits output schemas so strict clients can register the complete catalog; the generated developer manifest retains every canonical output schema for validation and typed SDK generation. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The unified server exposes 375 tools: 249 scraper, browser, workflow, billing, connected-service, and personal-assistant tools plus 126 durable-memory tools. The two retired customer credential mutations are no longer advertised. The scraper-side inventory includes complete Gmail selection, message, attachment, export, bulk-action, and Memory-import workflows; governed personal-assistant commands, messaging, approvals, grants, number setup, and execution readback; durable PAA starts and status; rendered site-content similarity; governed Local Sourcebook and Transparent Commons workflows; direct site-export reads; Editorial Reading Room and News Publisher templates; and production X-Ray setup, analytics, seven-model attribution, structured post-purchase surveys, reported impact, truthful view-evidence status, CRM policy and receipt, campaign, export, and scheduled-report tools. Provider setup remains absent until its authorization, ingestion, reconciliation, privacy, canary, cleanup, and deployment receipts are complete. Successful evidence-compiled Local Sourcebook revisions publish automatically to their canonical `localsourcebook.com` category profile and review URLs; administrator controls handle exceptional rejection or unpublishing. Release verification compares the exact local and hosted tool-name sets, not only the count.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The unified server exposes 376 tools: 250 scraper, browser, workflow, billing, connected-service, and personal-assistant tools plus 126 durable-memory tools. The two retired customer credential mutations are no longer advertised. The scraper-side inventory includes complete Gmail selection, message, attachment, export, bulk-action, and Memory-import workflows; governed personal-assistant commands, messaging, approvals, grants, number setup, and execution readback; durable PAA starts and status; rendered site-content similarity; governed Local Sourcebook and Transparent Commons workflows; direct site-export reads; Editorial Reading Room and News Publisher templates; and production X-Ray setup, analytics, seven-model attribution, structured post-purchase surveys, reported impact, truthful view-evidence status, CRM policy and receipt, campaign, export, and scheduled-report tools. Provider setup remains absent until its authorization, ingestion, reconciliation, privacy, canary, cleanup, and deployment receipts are complete. Successful evidence-compiled Local Sourcebook revisions publish automatically to their canonical `localsourcebook.com` category profile and review URLs; administrator controls handle exceptional rejection or unpublishing. Release verification compares the exact local and hosted tool-name sets, not only the count.
 
 For contract parity, stdio and MCPB memory calls invoke the matching public tool on the hosted MCP Scraper `/mcp` endpoint. The hosted aggregate runtime owns MCP Scraper-specific billing, scheduling, credential, and in-process cutover policy; its internal `/memory/mcp-call` bridge is a fallback to the standalone Memory service, not a second customer setup path. Existing direct Memory credentials remain compatible for one release, but all new customer setup uses the root endpoint and `MCP_SCRAPER_API_KEY`.
 
@@ -368,6 +369,8 @@ The `mcp-scraper` NPX stdio server also exposes saved reports as MCP resources:
 - `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `mcp-scraper` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
 - `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
 
+Hosted operators can isolate authorization state in a dedicated Turso/libSQL database without changing the public MCP tool catalog or API-key authentication. The secured store uses atomic authorization-code exchange and refresh rotation, keyed secret lookup, bounded encrypted replay receipts, authority epochs, and fail-closed maintenance behavior. Production migration and rollback are controlled data moves, not ordinary mode flips; see [MCP OAuth operations](docs/operations/mcp-oauth-runbook.md). Existing client setup and reconnect behavior are unchanged in Phase 1.
+
 Every web intelligence tool call made through `mcp-scraper` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. AI fan-out captures are always returned inline; only installed stdio/MCPB clients write optional `export=true` files under `MCP_SCRAPER_OUTPUT_DIR/fanout`, returning relative paths. Hosted clients always receive `exports: null`.
 
 ## Updating Existing Installs

package/dist/analytics-repository-J25XR5E7.js

@@ -0,0 +1 @@
+import{$ as L,$a as La,A as k,Aa as ka,B as l,Ba as la,C as m,Ca as ma,D as n,Da as na,E as o,Ea as oa,F as p,Fa as pa,G as q,Ga as qa,H as r,Ha as ra,I as s,Ia as sa,J as t,Ja as ta,K as u,Ka as ua,L as v,La as va,M as w,Ma as wa,N as x,Na as xa,O as y,Oa as ya,P as z,Pa as za,Q as A,Qa as Aa,R as B,Ra as Ba,S as C,Sa as Ca,T as D,Ta as Da,U as E,Ua as Ea,V as F,Va as Fa,W as G,Wa as Ga,X as H,Xa as Ha,Y as I,Ya as Ia,Z as J,Za as Ja,_ as K,_a as Ka,aa as M,ab as Ma,ba as N,bb as Na,ca as O,da as P,ea as Q,fa as R,ga as S,ha as T,ia as U,ja as V,ka as W,la as X,ma as Y,na as Z,oa as _,pa as $,q as a,qa as aa,r as b,ra as ba,s as c,sa as ca,t as d,ta as da,u as e,ua as ea,v as f,va as fa,w as g,wa as ga,x as h,xa as ha,y as i,ya as ia,z as j,za as ja}from"./chunk-645LQBE7.js";import"./chunk-D6IQJP64.js";import"./chunk-63T4W2LX.js";export{P as ANALYTICS_CONTENT_SORTS,a as AnalyticsRepositoryError,B as ENGAGED_SESSION_MS,A as MAX_ENGAGED_MS,N as analyticsAcquisition,l as analyticsBusinessMetrics,O as analyticsChannelBreakdown,R as analyticsContent,T as analyticsConversions,Ma as analyticsCsvCell,V as analyticsDimensions,S as analyticsEventCounts,m as analyticsForecast,Ja as analyticsHealth,x as analyticsIdentityPromotionAllowed,w as analyticsIdentityResolutionAllowed,L as analyticsOverview,U as analyticsPaths,M as analyticsTimeseries,H as appendAnalyticsAuthoritativeOutcomeVersion,ta as archiveAnalyticsActivationDestination,Y as archiveAnalyticsCampaignLink,ga as assignAnalyticsIdentityNode,fa as backfillAnalyticsConfirmedHistory,oa as claimAnalyticsCrmImportRows,Ea as claimAnalyticsFormDeliveryJobs,c as closeAnalyticsPool,pa as completeAnalyticsCrmImportRow,Ga as completeAnalyticsFormBridgeDelivery,Fa as completeAnalyticsFormDelivery,J as consumeAnalyticsSurveyInvite,ra as createAnalyticsActivationDestination,W as createAnalyticsCampaignLink,K as createAnalyticsConversion,ma as createAnalyticsCrmImport,Na as createAnalyticsExport,_ as createAnalyticsForm,o as createAnalyticsPixel,h as createAnalyticsSite,qa as deferAnalyticsCrmImportRow,Ha as deferAnalyticsFormDelivery,n as deleteAnalyticsSite,ea as deterministicAnalyticsCrmEntityId,ja as enrichAnalyticsExistingCrmIdentity,ua as getAnalyticsActivationDestinationConnectionRef,la as getAnalyticsPersonJourney,b as getAnalyticsPool,aa as getPublicAnalyticsForm,da as identityHmac,F as ingestAnalyticsEvents,G as insertAnalyticsRevenueSetupRevision,Ia as isAnalyticsFormPlacementApproved,ia as linkAnalyticsFormIdentity,ha as linkAnalyticsIdentityInTransaction,sa as listAnalyticsActivationDestinations,xa as listAnalyticsActivationReceipts,X as listAnalyticsCampaignLinks,na as listAnalyticsCrmImports,$ as listAnalyticsForms,t as listAnalyticsHostGroups,ka as listAnalyticsPeople,p as listAnalyticsPixels,i as listAnalyticsSites,d as migrateAnalytics,C as normalizeAnalyticsPath,D as normalizeAnalyticsUrl,Q as normalizeContentOptions,e as normalizeObservedHostname,za as pollAnalyticsActivationDiagnostics,u as prepareAnalyticsLinkerIssue,I as projectAnalyticsAuthoritativeConversion,y as projectAnalyticsPixelEventConsent,Aa as queueAnalyticsActivation,Ca as queueAnalyticsFormBridgeTransaction,Ba as queueAnalyticsFormDelivery,ba as recordAnalyticsFormSubmission,v as redeemAnalyticsLinkerRecord,Ka as refreshAnalyticsDailyRollups,La as refreshAnalyticsDailyRollupsIfDue,f as requireAnalyticsAccess,g as requireAnalyticsEditor,Z as resolveAnalyticsCampaignLink,z as resolveAnalyticsConfirmedActivationIdentity,ya as retryAnalyticsActivationJob,E as sanitizeAnalyticsProperties,ca as sanitizeClickIds,va as setAnalyticsActivationReadiness,r as setAnalyticsPixelDomainState,Da as sweepAnalyticsRestrictedRetention,wa as testAnalyticsActivationDestination,j as updateAnalyticsBusinessModel,q as updateAnalyticsPixel,k as upsertAnalyticsAdSpend,s as upsertAnalyticsHostGroup};

package/dist/bin/api-server.js

@@ -1,42 +1,3 @@
 #!/usr/bin/env node
-
-// bin/api-server.ts
-import { readFileSync } from "fs";
-function loadDotEnv() {
-  try {
-    for (const line of readFileSync(".env", "utf8").split("\n")) {
-      const eq = line.indexOf("=");
-      if (eq < 1 || line.trimStart().startsWith("#")) continue;
-      const k = line.slice(0, eq).trim();
-      if (!process.env[k]) process.env[k] = line.slice(eq + 1).trim();
-    }
-  } catch {
-  }
-}
-loadDotEnv();
-async function main() {
-  const [{ serve }, { app, personalAssistantProductionStartup }, { startWorker }, { migrate }] = await Promise.all([
-    import("@hono/node-server"),
-    import("../server-SQZ3A7SY.js"),
-    import("../worker-LDCAULWL.js"),
-    import("../db-Z34LPZNR.js")
-  ]);
-  const PORT = parseInt(process.env.PORT ?? "3001");
-  try {
-    await personalAssistantProductionStartup;
-    await migrate();
-    if (process.env.ANALYTICS_DATABASE_URL) {
-      const { migrateAnalytics } = await import("../analytics-repository-GGJJCVVP.js");
-      await migrateAnalytics();
-    }
-    startWorker();
-    serve({ fetch: app.fetch, port: PORT }, (info) => {
-      console.log(`[server] http://localhost:${info.port}`);
-      console.log(`[server] admin auth: ${process.env.ADMIN_KEY ? "configured" : "not configured"}`);
-    });
-  } catch (err) {
-    console.error("[startup] server preflight failed", err instanceof Error ? err.name : "unknown_error");
-    process.exit(1);
-  }
-}
-void main();
+import{readFileSync as a}from"fs";function c(){try{for(let t of a(".env","utf8").split(`
+`)){let o=t.indexOf("=");if(o<1||t.trimStart().startsWith("#"))continue;let e=t.slice(0,o).trim();process.env[e]||(process.env[e]=t.slice(o+1).trim())}}catch{}}c();async function p(){let[{serve:t},{app:o,personalAssistantProductionStartup:e},{startWorker:i},{migrate:n}]=await Promise.all([import("@hono/node-server"),import("../server-UEY5YQL6.js"),import("../worker-ZOTOKG56.js"),import("../db-JJWJLAVQ.js")]),s=parseInt(process.env.PORT??"3001");try{if(await e,await n(),process.env.ANALYTICS_DATABASE_URL){let{migrateAnalytics:r}=await import("../analytics-repository-J25XR5E7.js");await r()}i(),t({fetch:o.fetch,port:s},r=>{console.log(`[server] http://localhost:${r.port}`),console.log(`[server] admin auth: ${process.env.ADMIN_KEY?"configured":"not configured"}`)})}catch(r){console.error("[startup] server preflight failed",r instanceof Error?r.name:"unknown_error"),process.exit(1)}}p();