mcp-scraper 0.89.0 → 0.89.1

package/CHANGELOG.md

@@ -4,6 +4,22 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ## [Unreleased]
 
+## [0.89.1] - 2026-09-08
+
+### Changed
+
+- Added an operator-controlled path to isolate OAuth state in a dedicated Turso database with authority epochs, bounded cleanup, verified migration and rollback, and fail-closed maintenance behavior. Existing OAuth issuer, client setup, API-key authentication, and MCP tool contracts remain unchanged.
+- Bounded the root cron by admitting work lazily with explicit concurrency and deadline limits, so timed-out responses no longer eagerly start every maintenance task.
+
+### Fixed
+
+- Made authorization-code exchange and refresh-token rotation atomic and safely retryable after an uncertain database response, including deterministic replay of the one committed result under concurrent requests.
+- Accepted standards-compatible HTTPS, loopback, and reverse-domain native redirects while enforcing PKCE syntax, redirect matching, client binding, scope reduction, resource binding, and bounded dynamic registration.
+
+### Security
+
+- Replaced raw OAuth code and refresh-token storage in the canonical store with keyed digests and short-lived authenticated-encryption replay receipts, added key-retention checks, and kept destructive legacy-row cleanup behind a separate explicit gate.
+
 ## [0.89.0] - 2026-09-04
 
 ### Added
@@ -22,7 +38,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ### Fixed
 
-- Give each `reddit_thread` retrieval a 300-second end-to-end deadline, with two 60-second Option 1 attempts and two 60-second managed-browser backup attempts, instead of exhausting the full retry ladder in about 50 seconds. The MCP client now waits long enough to receive the endpoint's structured terminal result.
+- Give each `reddit_thread` retrieval a 300-second end-to-end deadline, with two 60-second Kernel attempts and two 60-second managed-browser backup attempts, instead of exhausting the full retry ladder in about 50 seconds. The MCP client now waits long enough to receive the endpoint's structured terminal result.
 
 ## [0.88.2] - 2026-09-02
 
@@ -59,19 +75,19 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ### Fixed
 
-- Kept Option 2 telemetry lookup off the Reddit response critical path and reallocated the saved time to 17-second backup attempts, so all four provider attempts can finish before production ends the request.
+- Kept Bright Data telemetry lookup off the Reddit response critical path and reallocated the saved time to 17-second backup attempts, so all four provider attempts can finish before production ends the request.
 
 ## [0.86.4] - 2026-09-02
 
 ### Fixed
 
-- Kept the complete two-primary, two-backup Reddit retry ladder inside the production request window by limiting Option 1 attempts to 8 seconds, Option 2 attempts to 14 seconds, and browser cleanup to 1 second.
+- Kept the complete two-primary, two-backup Reddit retry ladder inside the production request window by limiting Kernel attempts to 8 seconds, Bright Data attempts to 14 seconds, and browser cleanup to 1 second.
 
 ## [0.86.3] - 2026-09-02
 
 ### Fixed
 
-- Applied 45-second Option 1 and 35-second Option 2 deadlines to the complete Reddit browser-attempt lifecycle, and made known-thread primary attempts find and click the target through DuckDuckGo before the residential landing.
+- Applied 45-second Kernel and 35-second Bright Data deadlines to the complete Reddit browser-attempt lifecycle, and made known-thread primary attempts find and click the target through DuckDuckGo before the residential landing.
 
 ## [0.86.2] - 2026-09-02
 
@@ -220,13 +236,13 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ### Added
 
-- Added a Option 1-only Reddit workflow that searches DuckDuckGo with a `site:reddit.com` query, switches the same browser to a residential proxy before clicking the selected result, and reads modern Reddit posts plus bounded rendered-comment expansion through dedicated search, thread, and combined REST endpoints.
-- Added a bounded managed-browser backup for Reddit thread hydration after the primary Option 1 attempt fails or returns fewer than the semantic target, capped at 25 comments with measured bandwidth, duration, CAPTCHA, closure, and provider-cost telemetry.
+- Added a Kernel-only Reddit workflow that searches DuckDuckGo with a `site:reddit.com` query, switches the same browser to a residential proxy before clicking the selected result, and reads modern Reddit posts plus bounded rendered-comment expansion through dedicated search, thread, and combined REST endpoints.
+- Added a bounded managed-browser backup for Reddit thread hydration after the primary Kernel attempt fails or returns fewer than the semantic target, capped at 25 comments with measured bandwidth, duration, CAPTCHA, closure, and provider-cost telemetry.
 
 ### Changed
 
-- Routed the production `reddit_thread` and `reddit_trending` MCP tools through modern Reddit on Option 1 residential sessions, with DuckDuckGo site search for trend discovery; removed Google and old Reddit from their active execution path while preserving tool names, billing rates, bounded partial results, and refunds.
-- Cost probes now include Reddit Option 1 sessions and any managed-browser fallback bytes and cost in the same request receipt, and identify when the backup contributed to total cost.
+- Routed the production `reddit_thread` and `reddit_trending` MCP tools through modern Reddit on Kernel residential sessions, with DuckDuckGo site search for trend discovery; removed Google and old Reddit from their active execution path while preserving tool names, billing rates, bounded partial results, and refunds.
+- Cost probes now include Reddit Kernel sessions and any managed-browser fallback bytes and cost in the same request receipt, and identify when the backup contributed to total cost.
 
 ### Fixed
 
@@ -271,7 +287,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ### Fixed
 
-- Persisted per-control PAA dispatch and 0.7/1.0/1.4-second confirmation telemetry in durable checkpoints, exposed recent interaction and attempt correlation through MCP status, attached Option 2 session IDs immediately after browser launch, and finalized dangling attempt rows during lease recovery without blocking customer settlement.
+- Persisted per-control PAA dispatch and 0.7/1.0/1.4-second confirmation telemetry in durable checkpoints, exposed recent interaction and attempt correlation through MCP status, attached Bright Data session IDs immediately after browser launch, and finalized dangling attempt rows during lease recovery without blocking customer settlement.
 - Prevented inline style, script, and hidden DOM text inside Google answer containers from falsely confirming that PAA answer material loaded.
 - Routed canonical `/assistant` page loads to the web app and the redacted private Assistant readiness endpoint to the main API function, preventing production 404s after the 0.79.1 launch.
 
@@ -295,7 +311,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 - Added Scheduling as the canonical Personal Assistant setup surface, with connection readiness for Gmail, Calendar, Zoom, browser profiles, Memory, SMS, and email; exact schedule confirmation; approval and spend review; run history; and explicit watch/takeover states.
 - Added owner-scoped browser profiles that can hold multiple independently verified login bindings, while every browser schedule grant selects one exact profile, login, domain, and action set.
 - Added immutable schedule revisions, readiness receipts, append-only activation records, additive legacy schedule projection, and single-owner occurrence transition receipts so migration cannot silently infer browser authority or double-dispatch work.
-- Added Option 1 and private-Mac browser runtime boundaries with collision-resistant tenant namespaces, per-owner concurrency ceilings, bounded sessions, explicit and timeout cleanup, owner-qualified account deletion, and provider deletion readback.
+- Added Kernel and private-Mac browser runtime boundaries with collision-resistant tenant namespaces, per-owner concurrency ceilings, bounded sessions, explicit and timeout cleanup, owner-qualified account deletion, and provider deletion readback.
 - Added an owner-controlled Personal Assistant that brings SMS/MMS, Gmail, Google Calendar, Zoom, browser work, reminders, and Memory context packets into one governed workflow with immutable plans, approval checkpoints, spend limits, and durable receipts.
 - Added Twilio number discovery, owned-number attachment, purchase and registration previews, Messaging Service readiness, signed inbound and delivery webhooks, safe MMS ingestion, deterministic opt-out handling, single and reviewed bulk messaging, and reconciliation for unknown provider outcomes.
 - Added immutable, revisioned Memory context packets with source and attachment provenance, Gmail full-message imports, MMS media metadata, lifecycle controls, and readback verification against the selected vault.
@@ -332,7 +348,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 - Made `maxQuestions` an explicit target count rather than a traversal-depth control, with separate discovery and material-completeness diagnostics.
 - Preserved complete People Also Ask, AI Overview, and organic-result link provenance in JSON, structured MCP output, and CSV while classifying plain links and Google redirect links explicitly.
 - Resolved opaque Google `/goto` targets through bounded concurrent manual-redirect requests with active-browser interception as a fallback, without following publisher destinations and without dropping unresolved material.
-- Aligned the bounded PAA production-provider canary with the public `maxQuestions` contract and made Option 2 the default test provider.
+- Aligned the bounded PAA production-provider canary with the public `maxQuestions` contract and made Bright Data the default test provider.
 
 ### Fixed
 
@@ -540,7 +556,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 - Added portable `harvest_paa_start` and `harvest_paa_status` tools for durable long-running PAA research, with stable idempotency recovery, progress, attempt provenance, completeness, billing state, and bounded provider telemetry.
 - Added progressive PAA checkpoints that preserve and merge the best unique rows across browser retries and stale-job recovery instead of losing already captured questions when a provider session or caller is interrupted.
-- Added exact Option 2 browser-session identity, sanitized Session Logs enrichment, disconnect attribution, bandwidth usage telemetry, and retryable reconciliation without making provider telemetry a prerequisite for result delivery.
+- Added exact Bright Data browser-session identity, sanitized Session Logs enrichment, disconnect attribution, bandwidth usage telemetry, and retryable reconciliation without making provider telemetry a prerequisite for result delivery.
 
 ### Changed
 
@@ -1273,7 +1289,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ### Changed
 
-- PAA browser work now uses Option 1's co-located Playwright execution with stealth mode's default managed proxy and native browser metadata. Location is expressed only through Google UULE, CAPTCHA solver waiting is capped at 60 seconds, and a fresh session is allowed once only when no useful data was captured.
+- PAA browser work now uses Kernel's co-located Playwright execution with stealth mode's default managed proxy and native browser metadata. Location is expressed only through Google UULE, CAPTCHA solver waiting is capped at 60 seconds, and a fresh session is allowed once only when no useful data was captured.
 - PAA invocations stop browser work at 250 seconds inside the 280-second application budget, reserving 30 seconds for persistence, cleanup, and settlement. The legacy cron worker no longer claims Inngest-owned PAA jobs.
 
 ### Fixed
@@ -1509,7 +1525,7 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 
 ### Changed
 
-- `maps_search` now applies a transport ladder across its retry attempts so it can recover from Google soft-blocks instead of only retrying the same way. The first attempt is unchanged (Option 1's default stealth ISP proxy, direct navigation). Subsequent retries switch to direct egress and arrive at Google through a cross-site redirect (the combination that measurably clears blocks a cold navigation triggers); the final escalation attempt uses direct egress without the redirect and accepts any egress country. This only affects the `proxyMode: 'none'` default path and only its retries — a first-attempt success behaves exactly as before.
+- `maps_search` now applies a transport ladder across its retry attempts so it can recover from Google soft-blocks instead of only retrying the same way. The first attempt is unchanged (Kernel's default stealth ISP proxy, direct navigation). Subsequent retries switch to direct egress and arrive at Google through a cross-site redirect (the combination that measurably clears blocks a cold navigation triggers); the final escalation attempt uses direct egress without the redirect and accepts any egress country. This only affects the `proxyMode: 'none'` default path and only its retries — a first-attempt success behaves exactly as before.
 
 ## [0.32.1] - 2026-07-22
 
@@ -1901,3 +1917,5 @@ All notable changes to MCP Scraper are documented here. The format is based on [
 [0.11.0]: https://github.com/VilovietaSEO/mcp-scraper/releases/tag/v0.11.0
 
 [0.89.0]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.88.3...v0.89.0
+[Unreleased]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.89.1...HEAD
+[0.89.1]: https://github.com/VilovietaSEO/mcp-scraper/compare/v0.89.0...v0.89.1

package/README.md

@@ -175,7 +175,7 @@ Build the branded one-click bundle:
 npm run build:mcpb
 ```
 
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.89.0`, SHA-256 `3f2ecd89ea863e1ab15847aee70553755282cffb83c3ea9874beb57f0cf20927`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest.
+The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.89.1`, SHA-256 `f7ea38f87f0bb1bd73900f3817b47f923eb98c18428ca1b6f010d95f489aac74`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest.
 
 The MCPB install exposes every tool — web-intelligence plus all `browser_*` tools — through the one `mcp-scraper` server.
 
@@ -366,8 +366,10 @@ The `mcp-scraper` NPX stdio server also exposes saved reports as MCP resources:
 - `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`.
 - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
 - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
-- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `mcp-scraper` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `Option 1_BROWSER_PROFILE_NAME`, `Option 1_PROFILE_NAME`.
-- `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `Option 1_BROWSER_PROFILE_SAVE_CHANGES`, `Option 1_PROFILE_SAVE_CHANGES`.
+- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `mcp-scraper` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
+- `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
+
+Hosted operators can isolate authorization state in a dedicated Turso/libSQL database without changing the public MCP tool catalog or API-key authentication. The secured store uses atomic authorization-code exchange and refresh rotation, keyed secret lookup, bounded encrypted replay receipts, authority epochs, and fail-closed maintenance behavior. Production migration and rollback are controlled data moves, not ordinary mode flips; see [MCP OAuth operations](docs/operations/mcp-oauth-runbook.md). Existing client setup and reconnect behavior are unchanged in Phase 1.
 
 Every web intelligence tool call made through `mcp-scraper` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. AI fan-out captures are always returned inline; only installed stdio/MCPB clients write optional `export=true` files under `MCP_SCRAPER_OUTPUT_DIR/fanout`, returning relative paths. Hosted clients always receive `exports: null`.
 

package/dist/{analytics-repository-2BMT5JNE.js → analytics-repository-J25XR5E7.js}

@@ -1 +1 @@
-import{$ as L,$a as La,A as k,Aa as ka,B as l,Ba as la,C as m,Ca as ma,D as n,Da as na,E as o,Ea as oa,F as p,Fa as pa,G as q,Ga as qa,H as r,Ha as ra,I as s,Ia as sa,J as t,Ja as ta,K as u,Ka as ua,L as v,La as va,M as w,Ma as wa,N as x,Na as xa,O as y,Oa as ya,P as z,Pa as za,Q as A,Qa as Aa,R as B,Ra as Ba,S as C,Sa as Ca,T as D,Ta as Da,U as E,Ua as Ea,V as F,Va as Fa,W as G,Wa as Ga,X as H,Xa as Ha,Y as I,Ya as Ia,Z as J,Za as Ja,_ as K,_a as Ka,aa as M,ab as Ma,ba as N,bb as Na,ca as O,da as P,ea as Q,fa as R,ga as S,ha as T,ia as U,ja as V,ka as W,la as X,ma as Y,na as Z,oa as _,pa as $,q as a,qa as aa,r as b,ra as ba,s as c,sa as ca,t as d,ta as da,u as e,ua as ea,v as f,va as fa,w as g,wa as ga,x as h,xa as ha,y as i,ya as ia,z as j,za as ja}from"./chunk-WSCGYRWA.js";import"./chunk-M22MM4N4.js";import"./chunk-GMWKPIYX.js";export{P as ANALYTICS_CONTENT_SORTS,a as AnalyticsRepositoryError,B as ENGAGED_SESSION_MS,A as MAX_ENGAGED_MS,N as analyticsAcquisition,l as analyticsBusinessMetrics,O as analyticsChannelBreakdown,R as analyticsContent,T as analyticsConversions,Ma as analyticsCsvCell,V as analyticsDimensions,S as analyticsEventCounts,m as analyticsForecast,Ja as analyticsHealth,x as analyticsIdentityPromotionAllowed,w as analyticsIdentityResolutionAllowed,L as analyticsOverview,U as analyticsPaths,M as analyticsTimeseries,H as appendAnalyticsAuthoritativeOutcomeVersion,ta as archiveAnalyticsActivationDestination,Y as archiveAnalyticsCampaignLink,ga as assignAnalyticsIdentityNode,fa as backfillAnalyticsConfirmedHistory,oa as claimAnalyticsCrmImportRows,Ea as claimAnalyticsFormDeliveryJobs,c as closeAnalyticsPool,pa as completeAnalyticsCrmImportRow,Ga as completeAnalyticsFormBridgeDelivery,Fa as completeAnalyticsFormDelivery,J as consumeAnalyticsSurveyInvite,ra as createAnalyticsActivationDestination,W as createAnalyticsCampaignLink,K as createAnalyticsConversion,ma as createAnalyticsCrmImport,Na as createAnalyticsExport,_ as createAnalyticsForm,o as createAnalyticsPixel,h as createAnalyticsSite,qa as deferAnalyticsCrmImportRow,Ha as deferAnalyticsFormDelivery,n as deleteAnalyticsSite,ea as deterministicAnalyticsCrmEntityId,ja as enrichAnalyticsExistingCrmIdentity,ua as getAnalyticsActivationDestinationConnectionRef,la as getAnalyticsPersonJourney,b as getAnalyticsPool,aa as getPublicAnalyticsForm,da as identityHmac,F as ingestAnalyticsEvents,G as insertAnalyticsRevenueSetupRevision,Ia as isAnalyticsFormPlacementApproved,ia as linkAnalyticsFormIdentity,ha as linkAnalyticsIdentityInTransaction,sa as listAnalyticsActivationDestinations,xa as listAnalyticsActivationReceipts,X as listAnalyticsCampaignLinks,na as listAnalyticsCrmImports,$ as listAnalyticsForms,t as listAnalyticsHostGroups,ka as listAnalyticsPeople,p as listAnalyticsPixels,i as listAnalyticsSites,d as migrateAnalytics,C as normalizeAnalyticsPath,D as normalizeAnalyticsUrl,Q as normalizeContentOptions,e as normalizeObservedHostname,za as pollAnalyticsActivationDiagnostics,u as prepareAnalyticsLinkerIssue,I as projectAnalyticsAuthoritativeConversion,y as projectAnalyticsPixelEventConsent,Aa as queueAnalyticsActivation,Ca as queueAnalyticsFormBridgeTransaction,Ba as queueAnalyticsFormDelivery,ba as recordAnalyticsFormSubmission,v as redeemAnalyticsLinkerRecord,Ka as refreshAnalyticsDailyRollups,La as refreshAnalyticsDailyRollupsIfDue,f as requireAnalyticsAccess,g as requireAnalyticsEditor,Z as resolveAnalyticsCampaignLink,z as resolveAnalyticsConfirmedActivationIdentity,ya as retryAnalyticsActivationJob,E as sanitizeAnalyticsProperties,ca as sanitizeClickIds,va as setAnalyticsActivationReadiness,r as setAnalyticsPixelDomainState,Da as sweepAnalyticsRestrictedRetention,wa as testAnalyticsActivationDestination,j as updateAnalyticsBusinessModel,q as updateAnalyticsPixel,k as upsertAnalyticsAdSpend,s as upsertAnalyticsHostGroup};
+import{$ as L,$a as La,A as k,Aa as ka,B as l,Ba as la,C as m,Ca as ma,D as n,Da as na,E as o,Ea as oa,F as p,Fa as pa,G as q,Ga as qa,H as r,Ha as ra,I as s,Ia as sa,J as t,Ja as ta,K as u,Ka as ua,L as v,La as va,M as w,Ma as wa,N as x,Na as xa,O as y,Oa as ya,P as z,Pa as za,Q as A,Qa as Aa,R as B,Ra as Ba,S as C,Sa as Ca,T as D,Ta as Da,U as E,Ua as Ea,V as F,Va as Fa,W as G,Wa as Ga,X as H,Xa as Ha,Y as I,Ya as Ia,Z as J,Za as Ja,_ as K,_a as Ka,aa as M,ab as Ma,ba as N,bb as Na,ca as O,da as P,ea as Q,fa as R,ga as S,ha as T,ia as U,ja as V,ka as W,la as X,ma as Y,na as Z,oa as _,pa as $,q as a,qa as aa,r as b,ra as ba,s as c,sa as ca,t as d,ta as da,u as e,ua as ea,v as f,va as fa,w as g,wa as ga,x as h,xa as ha,y as i,ya as ia,z as j,za as ja}from"./chunk-645LQBE7.js";import"./chunk-D6IQJP64.js";import"./chunk-63T4W2LX.js";export{P as ANALYTICS_CONTENT_SORTS,a as AnalyticsRepositoryError,B as ENGAGED_SESSION_MS,A as MAX_ENGAGED_MS,N as analyticsAcquisition,l as analyticsBusinessMetrics,O as analyticsChannelBreakdown,R as analyticsContent,T as analyticsConversions,Ma as analyticsCsvCell,V as analyticsDimensions,S as analyticsEventCounts,m as analyticsForecast,Ja as analyticsHealth,x as analyticsIdentityPromotionAllowed,w as analyticsIdentityResolutionAllowed,L as analyticsOverview,U as analyticsPaths,M as analyticsTimeseries,H as appendAnalyticsAuthoritativeOutcomeVersion,ta as archiveAnalyticsActivationDestination,Y as archiveAnalyticsCampaignLink,ga as assignAnalyticsIdentityNode,fa as backfillAnalyticsConfirmedHistory,oa as claimAnalyticsCrmImportRows,Ea as claimAnalyticsFormDeliveryJobs,c as closeAnalyticsPool,pa as completeAnalyticsCrmImportRow,Ga as completeAnalyticsFormBridgeDelivery,Fa as completeAnalyticsFormDelivery,J as consumeAnalyticsSurveyInvite,ra as createAnalyticsActivationDestination,W as createAnalyticsCampaignLink,K as createAnalyticsConversion,ma as createAnalyticsCrmImport,Na as createAnalyticsExport,_ as createAnalyticsForm,o as createAnalyticsPixel,h as createAnalyticsSite,qa as deferAnalyticsCrmImportRow,Ha as deferAnalyticsFormDelivery,n as deleteAnalyticsSite,ea as deterministicAnalyticsCrmEntityId,ja as enrichAnalyticsExistingCrmIdentity,ua as getAnalyticsActivationDestinationConnectionRef,la as getAnalyticsPersonJourney,b as getAnalyticsPool,aa as getPublicAnalyticsForm,da as identityHmac,F as ingestAnalyticsEvents,G as insertAnalyticsRevenueSetupRevision,Ia as isAnalyticsFormPlacementApproved,ia as linkAnalyticsFormIdentity,ha as linkAnalyticsIdentityInTransaction,sa as listAnalyticsActivationDestinations,xa as listAnalyticsActivationReceipts,X as listAnalyticsCampaignLinks,na as listAnalyticsCrmImports,$ as listAnalyticsForms,t as listAnalyticsHostGroups,ka as listAnalyticsPeople,p as listAnalyticsPixels,i as listAnalyticsSites,d as migrateAnalytics,C as normalizeAnalyticsPath,D as normalizeAnalyticsUrl,Q as normalizeContentOptions,e as normalizeObservedHostname,za as pollAnalyticsActivationDiagnostics,u as prepareAnalyticsLinkerIssue,I as projectAnalyticsAuthoritativeConversion,y as projectAnalyticsPixelEventConsent,Aa as queueAnalyticsActivation,Ca as queueAnalyticsFormBridgeTransaction,Ba as queueAnalyticsFormDelivery,ba as recordAnalyticsFormSubmission,v as redeemAnalyticsLinkerRecord,Ka as refreshAnalyticsDailyRollups,La as refreshAnalyticsDailyRollupsIfDue,f as requireAnalyticsAccess,g as requireAnalyticsEditor,Z as resolveAnalyticsCampaignLink,z as resolveAnalyticsConfirmedActivationIdentity,ya as retryAnalyticsActivationJob,E as sanitizeAnalyticsProperties,ca as sanitizeClickIds,va as setAnalyticsActivationReadiness,r as setAnalyticsPixelDomainState,Da as sweepAnalyticsRestrictedRetention,wa as testAnalyticsActivationDestination,j as updateAnalyticsBusinessModel,q as updateAnalyticsPixel,k as upsertAnalyticsAdSpend,s as upsertAnalyticsHostGroup};

package/dist/bin/api-server.js

@@ -1,3 +1,3 @@
 #!/usr/bin/env node
 import{readFileSync as a}from"fs";function c(){try{for(let t of a(".env","utf8").split(`
-`)){let o=t.indexOf("=");if(o<1||t.trimStart().startsWith("#"))continue;let e=t.slice(0,o).trim();process.env[e]||(process.env[e]=t.slice(o+1).trim())}}catch{}}c();async function p(){let[{serve:t},{app:o,personalAssistantProductionStartup:e},{startWorker:i},{migrate:n}]=await Promise.all([import("@hono/node-server"),import("../server-RFR2A5UJ.js"),import("../worker-XUDSM3AL.js"),import("../db-F2MX63GI.js")]),s=parseInt(process.env.PORT??"3001");try{if(await e,await n(),process.env.ANALYTICS_DATABASE_URL){let{migrateAnalytics:r}=await import("../analytics-repository-2BMT5JNE.js");await r()}i(),t({fetch:o.fetch,port:s},r=>{console.log(`[server] http://localhost:${r.port}`),console.log(`[server] admin auth: ${process.env.ADMIN_KEY?"configured":"not configured"}`)})}catch(r){console.error("[startup] server preflight failed",r instanceof Error?r.name:"unknown_error"),process.exit(1)}}p();
+`)){let o=t.indexOf("=");if(o<1||t.trimStart().startsWith("#"))continue;let e=t.slice(0,o).trim();process.env[e]||(process.env[e]=t.slice(o+1).trim())}}catch{}}c();async function p(){let[{serve:t},{app:o,personalAssistantProductionStartup:e},{startWorker:i},{migrate:n}]=await Promise.all([import("@hono/node-server"),import("../server-UEY5YQL6.js"),import("../worker-ZOTOKG56.js"),import("../db-JJWJLAVQ.js")]),s=parseInt(process.env.PORT??"3001");try{if(await e,await n(),process.env.ANALYTICS_DATABASE_URL){let{migrateAnalytics:r}=await import("../analytics-repository-J25XR5E7.js");await r()}i(),t({fetch:o.fetch,port:s},r=>{console.log(`[server] http://localhost:${r.port}`),console.log(`[server] admin auth: ${process.env.ADMIN_KEY?"configured":"not configured"}`)})}catch(r){console.error("[startup] server preflight failed",r instanceof Error?r.name:"unknown_error"),process.exit(1)}}p();

package/dist/bin/mcp-scraper-cli.js

@@ -1,5 +1,5 @@
 #!/usr/bin/env node
-import{b as E,c as N,d as M,e as L,f as T,j as U}from"../chunk-XLWNEVUZ.js";import"../chunk-KJQXUZ4Y.js";import"../chunk-3GP5CYZX.js";import{g as v,j as b,k as D,l as K,m as H}from"../chunk-WO3N5FH2.js";import"../chunk-HE45FFBU.js";import{a as P}from"../chunk-Z2QGQJS2.js";import{Command as he}from"commander";import{spawn as ne}from"child_process";import{mkdir as ke,writeFile as Pe}from"fs/promises";import{basename as Ce,join as Z}from"path";function se(e){return e.apiKey?.trim()||"sk_live_your_key"}function ce(e){return e.packageSpec?.trim()||"mcp-scraper@latest"}function A(e={}){return["-y","--package",ce(e),"mcp-scraper"]}function pe(e){let n={MCP_SCRAPER_API_KEY:se(e)},c=e.browserProfileName?.trim();return c&&(n.BROWSER_AGENT_PROFILE_NAME=c),e.browserProfileSaveChanges===!0&&(n.BROWSER_AGENT_PROFILE_SAVE_CHANGES="true"),n}function q(){return["mcp","remove","mcp-scraper","-s","user"]}function J(){return["mcp","get","mcp-scraper"]}function B(e){let n=e.match(/^\s*Command:\s*(.+?)\s*$/m)?.[1];if(!n)return null;let c=e.match(/^\s*Args:\s*(.*?)\s*$/m)?.[1]??"",i=c.length?c.split(/\s+/):[],p={},u=e.split(/^\s*Environment:\s*$/m)[1];if(u)for(let a of u.split(`
+import{b as E,c as N,d as M,e as L,f as T,j as U}from"../chunk-XLWNEVUZ.js";import"../chunk-KJQXUZ4Y.js";import"../chunk-3GP5CYZX.js";import{g as v,j as b,k as D,l as K,m as H}from"../chunk-WO3N5FH2.js";import"../chunk-HE45FFBU.js";import{a as P}from"../chunk-5H3MCTTP.js";import{Command as he}from"commander";import{spawn as ne}from"child_process";import{mkdir as ke,writeFile as Pe}from"fs/promises";import{basename as Ce,join as Z}from"path";function se(e){return e.apiKey?.trim()||"sk_live_your_key"}function ce(e){return e.packageSpec?.trim()||"mcp-scraper@latest"}function A(e={}){return["-y","--package",ce(e),"mcp-scraper"]}function pe(e){let n={MCP_SCRAPER_API_KEY:se(e)},c=e.browserProfileName?.trim();return c&&(n.BROWSER_AGENT_PROFILE_NAME=c),e.browserProfileSaveChanges===!0&&(n.BROWSER_AGENT_PROFILE_SAVE_CHANGES="true"),n}function q(){return["mcp","remove","mcp-scraper","-s","user"]}function J(){return["mcp","get","mcp-scraper"]}function B(e){let n=e.match(/^\s*Command:\s*(.+?)\s*$/m)?.[1];if(!n)return null;let c=e.match(/^\s*Args:\s*(.*?)\s*$/m)?.[1]??"",i=c.length?c.split(/\s+/):[],p={},u=e.split(/^\s*Environment:\s*$/m)[1];if(u)for(let a of u.split(`
 `)){let l=a.match(/^\s{2,}([A-Za-z_][A-Za-z0-9_]*)=(.*)$/);if(!l){if(a.trim().length&&!/^\s{2,}/.test(a))break;continue}p[l[1]]=l[2]}return{command:n,args:i,env:p}}function j(e){let n=["mcp","add","mcp-scraper","--scope","user"];for(let[c,i]of Object.entries(e.env))n.push("--env",`${c}=${i}`);return n.push("--",e.command,...e.args),n}function G(e={}){let n=["mcp","add","mcp-scraper","--scope","user"];for(let[c,i]of Object.entries(pe(e)))n.push("--env",`${c}=${i}`);return n.push("--","npx",...A(e)),n}function O(e){if(e==="claude-code")return"claude";if(e==="claude"||D.hosts.some(n=>n.id===e))return e;throw new Error('Unknown host "'+e+'". Use: codex, claude, claude-code, claude-desktop, cursor, windsurf, cline, or user-action-only')}function ue(e){return K(e==="claude"?"claude-code":e)}function W(e,n={}){let c=O(e),i=ue(c),p="Restart the MCP client so it starts a fresh npx process.",u='MCP_SCRAPER_API_KEY="$MCP_SCRAPER_API_KEY" npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude --apply',a=`X-Ray install protocol: ${v} (${b})`;return c==="codex"?["# Codex MCP config",a,i.exactConfig,"",`Continuation: ${i.continuation}`,`Rollback: ${i.rollback}`,"",p].join(`
 `):c==="claude"?["# Claude Code command",a,i.exactConfig,"","# One-command Claude Code setup",u,"",`Continuation: ${i.continuation}`,`Rollback: ${i.rollback}`,"",p].join(`
 `):c==="claude-desktop"?["# Claude Desktop config",a,i.exactConfig,"","Desktop Extension: https://mcpscraper.dev/downloads/mcp-scraper.mcpb",`Continuation: ${i.continuation}`,`Rollback: ${i.rollback}`,p].join(`

package/dist/bin/mcp-scraper-core.js

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-import{a as e}from"../chunk-L4CGLFPU.js";import"../chunk-PUHFVA7P.js";import"../chunk-W2BVJ7S2.js";import"../chunk-TMB56NCA.js";import"../chunk-HUV2WTRW.js";import"../chunk-X54CQLK2.js";import"../chunk-MZN4U5BL.js";import"../chunk-4FROKQJN.js";import"../chunk-MASR22K4.js";import"../chunk-CCYWSJNG.js";import"../chunk-WO3N5FH2.js";import"../chunk-HE45FFBU.js";import"../chunk-Z2QGQJS2.js";import"../chunk-GMWKPIYX.js";var _=["harvest_paa","search_serp","extract_url","diff_page","map_site_urls","map_wayback_snapshots","extract_site","analyze_site_similarity","audit_site","check_site_export","site_export_read","site_export_image","archive_read","youtube_harvest","youtube_transcribe","facebook_page_intel","facebook_ad_search","reddit_thread","reddit_trending","video_frame_analysis","video_frame_analysis_status","facebook_ad_transcribe","google_ads_search","google_ads_page_intel","google_ads_transcribe","facebook_video_transcribe","instagram_profile_content","instagram_media_download","maps_place_intel","maps_search","trustpilot_reviews","g2_reviews","capture_serp_snapshot","capture_serp_page_snapshots"];e({toolsets:new Set(["paa","serp"]),allowedToolNames:_});
+import{a as e}from"../chunk-YUPLOOTB.js";import"../chunk-MEM4D2QN.js";import"../chunk-W2BVJ7S2.js";import"../chunk-TMB56NCA.js";import"../chunk-HUV2WTRW.js";import"../chunk-FXEWHK54.js";import"../chunk-MZN4U5BL.js";import"../chunk-4FROKQJN.js";import"../chunk-RYNHNEF3.js";import"../chunk-CCYWSJNG.js";import"../chunk-WO3N5FH2.js";import"../chunk-HE45FFBU.js";import"../chunk-5H3MCTTP.js";import"../chunk-63T4W2LX.js";var _=["harvest_paa","search_serp","extract_url","diff_page","map_site_urls","map_wayback_snapshots","extract_site","analyze_site_similarity","audit_site","check_site_export","site_export_read","site_export_image","archive_read","youtube_harvest","youtube_transcribe","facebook_page_intel","facebook_ad_search","reddit_thread","reddit_trending","video_frame_analysis","video_frame_analysis_status","facebook_ad_transcribe","google_ads_search","google_ads_page_intel","google_ads_transcribe","facebook_video_transcribe","instagram_profile_content","instagram_media_download","maps_place_intel","maps_search","trustpilot_reviews","g2_reviews","capture_serp_snapshot","capture_serp_page_snapshots"];e({toolsets:new Set(["paa","serp"]),allowedToolNames:_});

package/dist/bin/mcp-scraper-install.js

@@ -1,3 +1,3 @@
 #!/usr/bin/env node
-import{a as s}from"../chunk-CCYWSJNG.js";import{a as e}from"../chunk-Z2QGQJS2.js";var r=process.argv.includes("--no-color")||process.env.NO_COLOR!==void 0||process.env.FORCE_COLOR==="0"||!process.stdout.isTTY,n=process.argv.includes("--help")||process.argv.includes("-h");n&&(process.stdout.write(["Usage: mcp-scraper-install [--no-color]","","Prints the branded MCP Scraper terminal install card and copyable install commands.","mcp-scraper prints the same card in a human terminal and runs as the MCP stdio server in clients.",""].join(`
+import{a as s}from"../chunk-CCYWSJNG.js";import{a as e}from"../chunk-5H3MCTTP.js";var r=process.argv.includes("--no-color")||process.env.NO_COLOR!==void 0||process.env.FORCE_COLOR==="0"||!process.stdout.isTTY,n=process.argv.includes("--help")||process.argv.includes("-h");n&&(process.stdout.write(["Usage: mcp-scraper-install [--no-color]","","Prints the branded MCP Scraper terminal install card and copyable install commands.","mcp-scraper prints the same card in a human terminal and runs as the MCP stdio server in clients.",""].join(`
 `)),process.exit(0));process.stdout.write(s({version:e,color:!r,apiKeyConfigured:!!process.env.MCP_SCRAPER_API_KEY?.trim()}));

package/dist/bin/mcp-stdio-server.js

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-import{a as r}from"../chunk-L4CGLFPU.js";import"../chunk-PUHFVA7P.js";import"../chunk-W2BVJ7S2.js";import"../chunk-TMB56NCA.js";import"../chunk-HUV2WTRW.js";import"../chunk-X54CQLK2.js";import"../chunk-MZN4U5BL.js";import"../chunk-4FROKQJN.js";import"../chunk-MASR22K4.js";import"../chunk-CCYWSJNG.js";import"../chunk-WO3N5FH2.js";import"../chunk-HE45FFBU.js";import"../chunk-Z2QGQJS2.js";import"../chunk-GMWKPIYX.js";r();
+import{a as r}from"../chunk-YUPLOOTB.js";import"../chunk-MEM4D2QN.js";import"../chunk-W2BVJ7S2.js";import"../chunk-TMB56NCA.js";import"../chunk-HUV2WTRW.js";import"../chunk-FXEWHK54.js";import"../chunk-MZN4U5BL.js";import"../chunk-4FROKQJN.js";import"../chunk-RYNHNEF3.js";import"../chunk-CCYWSJNG.js";import"../chunk-WO3N5FH2.js";import"../chunk-HE45FFBU.js";import"../chunk-5H3MCTTP.js";import"../chunk-63T4W2LX.js";r();

package/dist/bin/paa-harvest.js

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-import{u as t}from"../chunk-7YGVI5J4.js";import"../chunk-QPWPR5XG.js";import{a as r}from"../chunk-4FROKQJN.js";import"../chunk-MASR22K4.js";import"../chunk-3GP5CYZX.js";import"../chunk-GMWKPIYX.js";import{Command as s,Option as a}from"commander";var i=new s;i.name("paa-harvest").description("Recursively extract Google People Also Ask questions").requiredOption("-q, --query <query>","Seed query").option("-l, --location <location>",'Location name (e.g. "austin" or "Austin,Texas,United States")').option("--gl <gl>","Google country code","us").option("--hl <hl>","Google language code","en").option("-d, --depth <depth>","BFS depth (1-30)","3").option("-m, --max-questions <n>","Max questions to harvest","100").option("-o, --output <dir>","Output directory","./paa-output").option("-f, --format <format>","Output format: json, csv, or both","both").option("--headless","Run browser in headless mode",!1).option("--profile <dir>","Persistent browser profile directory").option("--proxy <url>","Proxy server URL").option("--browser-api-key <key>","Browser service API key (or set BROWSER_SERVICE_API_KEY env var)").addOption(new a("--\u006b\u0065\u0072\u006e\u0065\u006c-api-key <key>").hideHelp()).action(async e=>{try{let o=await t({query:e.query,location:e.location,gl:e.gl,hl:e.hl,depth:parseInt(e.depth,10),maxQuestions:parseInt(e.maxQuestions,10),outputDir:e.output,format:e.format,headless:e.headless,profileDir:e.profile,proxy:e.proxy,\u006b\u0065\u0072\u006e\u0065\u006cApiKey:e.browserApiKey??e.\u006b\u0065\u0072\u006e\u0065\u006cApiKey??r()});console.log(JSON.stringify({totalQuestions:o.totalQuestions,outputDir:o.stats.seed}))}catch(o){console.error(o instanceof Error?o.message:String(o)),process.exit(1)}});async function n(){await i.parseAsync()}n();
+import{u as t}from"../chunk-6QO4F2KC.js";import"../chunk-QPWPR5XG.js";import{a as r}from"../chunk-4FROKQJN.js";import"../chunk-RYNHNEF3.js";import"../chunk-3GP5CYZX.js";import"../chunk-63T4W2LX.js";import{Command as s,Option as a}from"commander";var i=new s;i.name("paa-harvest").description("Recursively extract Google People Also Ask questions").requiredOption("-q, --query <query>","Seed query").option("-l, --location <location>",'Location name (e.g. "austin" or "Austin,Texas,United States")').option("--gl <gl>","Google country code","us").option("--hl <hl>","Google language code","en").option("-d, --depth <depth>","BFS depth (1-30)","3").option("-m, --max-questions <n>","Max questions to harvest","100").option("-o, --output <dir>","Output directory","./paa-output").option("-f, --format <format>","Output format: json, csv, or both","both").option("--headless","Run browser in headless mode",!1).option("--profile <dir>","Persistent browser profile directory").option("--proxy <url>","Proxy server URL").option("--browser-api-key <key>","Browser service API key (or set BROWSER_SERVICE_API_KEY env var)").addOption(new a("--kernel-api-key <key>").hideHelp()).action(async e=>{try{let o=await t({query:e.query,location:e.location,gl:e.gl,hl:e.hl,depth:parseInt(e.depth,10),maxQuestions:parseInt(e.maxQuestions,10),outputDir:e.output,format:e.format,headless:e.headless,profileDir:e.profile,proxy:e.proxy,kernelApiKey:e.browserApiKey??e.kernelApiKey??r()});console.log(JSON.stringify({totalQuestions:o.totalQuestions,outputDir:o.stats.seed}))}catch(o){console.error(o instanceof Error?o.message:String(o)),process.exit(1)}});async function n(){await i.parseAsync()}n();

package/dist/chunk-3GP5CYZX.js

@@ -1 +1 @@
-import{z as e}from"zod";var r="none",i="none",u=e.object({query:e.string().min(1),location:e.string().optional(),gl:e.string().length(2).default("us"),hl:e.string().length(2).default("en"),device:e.enum(["desktop","mobile"]).default("desktop"),proxyMode:e.enum(["location","configured","none"]).default(r),proxyZip:e.string().regex(/^\d{5}$/).optional(),keepDefaultProxy:e.boolean().optional(),requireUsEgress:e.boolean().optional(),maxAttempts:e.number().int().min(1).max(12).optional(),debug:e.boolean().default(!1),depth:e.number().int().min(1).max(30).default(3),maxQuestions:e.number().int().min(1).max(1e3).default(100),headless:e.boolean().default(!1),profileDir:e.string().optional(),proxy:e.string().url().optional(),\u006b\u0065\u0072\u006e\u0065\u006cApiKey:e.string().optional(),\u006b\u0065\u0072\u006e\u0065\u006cProxyId:e.string().optional(),\u006b\u0065\u0072\u006e\u0065\u006cProfileName:e.string().optional(),\u006b\u0065\u0072\u006e\u0065\u006cProfileSaveChanges:e.boolean().optional(),\u006b\u0065\u0072\u006e\u0065\u006cStealth:e.boolean().optional(),serpIdentity:e.string().regex(/^[a-z0-9][a-z0-9_-]{0,63}$/).optional(),\u006b\u0065\u0072\u006e\u0065\u006cProxyResolution:e.unknown().optional(),outputDir:e.string().default("./paa-output"),format:e.enum(["json","csv","both"]).default("both"),serpOnly:e.boolean().default(!1),questionsOnly:e.boolean().default(!1),questionGrowthRecoveryRounds:e.number().int().min(0).max(4).default(2),pages:e.number().int().min(1).max(2).default(1),includeAllSerpFeatures:e.boolean().default(!1),includeLocalPack:e.boolean().default(!1),includeForums:e.boolean().default(!1),includeVideos:e.boolean().default(!1),includeAiOverview:e.boolean().default(!1),includeWhatPeopleSaying:e.boolean().default(!1),includeShortVideos:e.boolean().default(!1),recency:e.enum(["day","week","month","year"]).optional(),softDeadlineMs:e.number().optional()});function d(t){switch(t){case"day":return"qdr:d";case"week":return"qdr:w";case"month":return"qdr:m";case"year":return"qdr:y";default:return}}var c=e.object({businessName:e.string().min(1),location:e.string().min(1),gl:e.string().length(2).default("us"),hl:e.string().length(2).default("en"),includeReviews:e.boolean().default(!1),maxReviews:e.number().int().min(1).max(500).default(50),includeServices:e.boolean().default(!1),includeImages:e.boolean().default(!1),imageScope:e.enum(["owner","all"]).default("all"),maxImages:e.number().int().min(1).max(250).default(100),maxInlineImages:e.number().int().min(0).max(5).default(3),\u006b\u0065\u0072\u006e\u0065\u006cApiKey:e.string().optional(),\u006b\u0065\u0072\u006e\u0065\u006cProxyId:e.string().optional(),headless:e.boolean().default(!0)}),g=e.object({query:e.string().min(1),location:e.string().optional(),gl:e.string().length(2).default("us"),hl:e.string().length(2).default("en"),maxResults:e.number().int().min(1).max(50).default(10),includeServices:e.boolean().default(!1),proxyMode:e.enum(["location","configured","none"]).default(i),proxyZip:e.string().regex(/^\d{5}$/).optional(),serpRedirect:e.boolean().optional(),forceDirectEgress:e.boolean().optional(),debug:e.boolean().default(!1),\u006b\u0065\u0072\u006e\u0065\u006cApiKey:e.string().optional(),\u006b\u0065\u0072\u006e\u0065\u006cProxyId:e.string().optional(),\u006b\u0065\u0072\u006e\u0065\u006cProxyResolution:e.unknown().optional(),headless:e.boolean().default(!0)}),p=e.object({questionId:e.string().min(1),googleLinkId:e.string().nullable(),googleEvidenceId:e.string().nullable(),question:e.string().min(1),answer:e.string().optional(),sourceTitle:e.string().optional(),sourceSite:e.string().optional(),sourceCite:e.string().optional(),sources:e.array(e.object({title:e.string().nullable(),site:e.string().nullable(),url:e.string().min(1),rawUrl:e.string().min(1),resolvedUrl:e.string().nullable(),linkType:e.enum(["plain","google_url_redirect","google_goto_redirect"]),resolutionStatus:e.enum(["not_needed","resolved","unresolved","rejected"])})).default([])}),m=e.object({name:e.string().nullable(),rating:e.string().nullable(),reviewCount:e.string().nullable(),category:e.string().nullable(),address:e.string().nullable(),hoursSummary:e.string().nullable(),phone:e.string().nullable(),phoneDisplay:e.string().nullable(),website:e.string().nullable(),plusCode:e.string().nullable(),bookingUrl:e.string().nullable()}),b=e.object({day:e.string(),hours:e.string()}),f=e.object({reviewHistogram:e.array(e.object({stars:e.number(),count:e.string()})),reviewTopics:e.array(e.object({label:e.string(),count:e.string()}))}),y=e.object({reviewId:e.string(),author:e.string().nullable(),stars:e.string().nullable(),date:e.string().nullable(),text:e.string().nullable(),ownerResponse:e.string().nullable()}),h=e.object({section:e.string(),attribute:e.string()});async function w(t){let o=process.env.MCP_SCRAPER_API_KEY?.trim(),a=(process.env.MCP_SCRAPER_BASE_URL??"https://mcpscraper.dev").replace(/\/$/,"");if(o)try{let n=await fetch(`${a}/memory/mcp-call`,{method:"POST",headers:{"content-type":"application/json","x-api-key":o},body:JSON.stringify({toolName:"libraryIngestTool",args:t})}),l=await n.json().catch(()=>null);(!n.ok||l?.ok===!1)&&console.warn("[memory-library-sink] ingest not accepted:",n.status,l?.error??"")}catch(n){console.warn("[memory-library-sink] ingest failed:",n?.message)}}export{r as a,i as b,u as c,d,c as e,g as f,p as g,m as h,b as i,f as j,h as k,w as l};
+import{z as e}from"zod";var r="none",i="none",u=e.object({query:e.string().min(1),location:e.string().optional(),gl:e.string().length(2).default("us"),hl:e.string().length(2).default("en"),device:e.enum(["desktop","mobile"]).default("desktop"),proxyMode:e.enum(["location","configured","none"]).default(r),proxyZip:e.string().regex(/^\d{5}$/).optional(),keepDefaultProxy:e.boolean().optional(),requireUsEgress:e.boolean().optional(),maxAttempts:e.number().int().min(1).max(12).optional(),debug:e.boolean().default(!1),depth:e.number().int().min(1).max(30).default(3),maxQuestions:e.number().int().min(1).max(1e3).default(100),headless:e.boolean().default(!1),profileDir:e.string().optional(),proxy:e.string().url().optional(),kernelApiKey:e.string().optional(),kernelProxyId:e.string().optional(),kernelProfileName:e.string().optional(),kernelProfileSaveChanges:e.boolean().optional(),kernelStealth:e.boolean().optional(),serpIdentity:e.string().regex(/^[a-z0-9][a-z0-9_-]{0,63}$/).optional(),kernelProxyResolution:e.unknown().optional(),outputDir:e.string().default("./paa-output"),format:e.enum(["json","csv","both"]).default("both"),serpOnly:e.boolean().default(!1),questionsOnly:e.boolean().default(!1),questionGrowthRecoveryRounds:e.number().int().min(0).max(4).default(2),pages:e.number().int().min(1).max(2).default(1),includeAllSerpFeatures:e.boolean().default(!1),includeLocalPack:e.boolean().default(!1),includeForums:e.boolean().default(!1),includeVideos:e.boolean().default(!1),includeAiOverview:e.boolean().default(!1),includeWhatPeopleSaying:e.boolean().default(!1),includeShortVideos:e.boolean().default(!1),recency:e.enum(["day","week","month","year"]).optional(),softDeadlineMs:e.number().optional()});function d(t){switch(t){case"day":return"qdr:d";case"week":return"qdr:w";case"month":return"qdr:m";case"year":return"qdr:y";default:return}}var c=e.object({businessName:e.string().min(1),location:e.string().min(1),gl:e.string().length(2).default("us"),hl:e.string().length(2).default("en"),includeReviews:e.boolean().default(!1),maxReviews:e.number().int().min(1).max(500).default(50),includeServices:e.boolean().default(!1),includeImages:e.boolean().default(!1),imageScope:e.enum(["owner","all"]).default("all"),maxImages:e.number().int().min(1).max(250).default(100),maxInlineImages:e.number().int().min(0).max(5).default(3),kernelApiKey:e.string().optional(),kernelProxyId:e.string().optional(),headless:e.boolean().default(!0)}),g=e.object({query:e.string().min(1),location:e.string().optional(),gl:e.string().length(2).default("us"),hl:e.string().length(2).default("en"),maxResults:e.number().int().min(1).max(50).default(10),includeServices:e.boolean().default(!1),proxyMode:e.enum(["location","configured","none"]).default(i),proxyZip:e.string().regex(/^\d{5}$/).optional(),serpRedirect:e.boolean().optional(),forceDirectEgress:e.boolean().optional(),debug:e.boolean().default(!1),kernelApiKey:e.string().optional(),kernelProxyId:e.string().optional(),kernelProxyResolution:e.unknown().optional(),headless:e.boolean().default(!0)}),p=e.object({questionId:e.string().min(1),googleLinkId:e.string().nullable(),googleEvidenceId:e.string().nullable(),question:e.string().min(1),answer:e.string().optional(),sourceTitle:e.string().optional(),sourceSite:e.string().optional(),sourceCite:e.string().optional(),sources:e.array(e.object({title:e.string().nullable(),site:e.string().nullable(),url:e.string().min(1),rawUrl:e.string().min(1),resolvedUrl:e.string().nullable(),linkType:e.enum(["plain","google_url_redirect","google_goto_redirect"]),resolutionStatus:e.enum(["not_needed","resolved","unresolved","rejected"])})).default([])}),m=e.object({name:e.string().nullable(),rating:e.string().nullable(),reviewCount:e.string().nullable(),category:e.string().nullable(),address:e.string().nullable(),hoursSummary:e.string().nullable(),phone:e.string().nullable(),phoneDisplay:e.string().nullable(),website:e.string().nullable(),plusCode:e.string().nullable(),bookingUrl:e.string().nullable()}),b=e.object({day:e.string(),hours:e.string()}),f=e.object({reviewHistogram:e.array(e.object({stars:e.number(),count:e.string()})),reviewTopics:e.array(e.object({label:e.string(),count:e.string()}))}),y=e.object({reviewId:e.string(),author:e.string().nullable(),stars:e.string().nullable(),date:e.string().nullable(),text:e.string().nullable(),ownerResponse:e.string().nullable()}),h=e.object({section:e.string(),attribute:e.string()});async function w(t){let o=process.env.MCP_SCRAPER_API_KEY?.trim(),a=(process.env.MCP_SCRAPER_BASE_URL??"https://mcpscraper.dev").replace(/\/$/,"");if(o)try{let n=await fetch(`${a}/memory/mcp-call`,{method:"POST",headers:{"content-type":"application/json","x-api-key":o},body:JSON.stringify({toolName:"libraryIngestTool",args:t})}),l=await n.json().catch(()=>null);(!n.ok||l?.ok===!1)&&console.warn("[memory-library-sink] ingest not accepted:",n.status,l?.error??"")}catch(n){console.warn("[memory-library-sink] ingest failed:",n?.message)}}export{r as a,i as b,u as c,d,c as e,g as f,p as g,m as h,b as i,f as j,h as k,w as l};

package/dist/chunk-4FROKQJN.js

@@ -1 +1 @@
-function n(){return(process.env.BROWSER_SERVICE_API_KEY??process.env.\u004b\u0045\u0052\u004e\u0045\u004c_API_KEY)?.trim()||void 0}function r(){return(process.env.BROWSER_AGENT_PROFILE_NAME??process.env.BROWSER_SERVICE_PROFILE_NAME??process.env.\u004b\u0045\u0052\u004e\u0045\u004c_BROWSER_PROFILE_NAME??process.env.\u004b\u0045\u0052\u004e\u0045\u004c_PROFILE_NAME)?.trim()||void 0}function E(){let e=(process.env.BROWSER_AGENT_PROFILE_SAVE_CHANGES??process.env.BROWSER_SERVICE_PROFILE_SAVE_CHANGES??process.env.\u004b\u0045\u0052\u004e\u0045\u004c_BROWSER_PROFILE_SAVE_CHANGES??process.env.\u004b\u0045\u0052\u004e\u0045\u004c_PROFILE_SAVE_CHANGES)?.trim().toLowerCase();if(e){if(["1","true","yes","on"].includes(e))return!0;if(["0","false","no","off"].includes(e))return!1}}export{n as a,r as b,E as c};
+function n(){return(process.env.BROWSER_SERVICE_API_KEY??process.env.KERNEL_API_KEY)?.trim()||void 0}function r(){return(process.env.BROWSER_AGENT_PROFILE_NAME??process.env.BROWSER_SERVICE_PROFILE_NAME??process.env.KERNEL_BROWSER_PROFILE_NAME??process.env.KERNEL_PROFILE_NAME)?.trim()||void 0}function E(){let e=(process.env.BROWSER_AGENT_PROFILE_SAVE_CHANGES??process.env.BROWSER_SERVICE_PROFILE_SAVE_CHANGES??process.env.KERNEL_BROWSER_PROFILE_SAVE_CHANGES??process.env.KERNEL_PROFILE_SAVE_CHANGES)?.trim().toLowerCase();if(e){if(["1","true","yes","on"].includes(e))return!0;if(["0","false","no","off"].includes(e))return!1}}export{n as a,r as b,E as c};

package/dist/{chunk-TXENITMS.js → chunk-57AHYL2O.js}

@@ -1,4 +1,4 @@
-import{a as N}from"./chunk-YQZGZBB4.js";import{b as L,c as P}from"./chunk-HE45FFBU.js";import{h as $,s as x}from"./chunk-GMWKPIYX.js";import{createWriteStream as Z,mkdirSync as ee,rmSync as te}from"fs";import{homedir as re}from"os";import{join as I,extname as k,basename as B}from"path";import{pipeline as ne}from"stream/promises";import{Readable as oe,Transform as ie}from"stream";var se=["doubleclick.net","googlesyndication.com","googletagmanager.com","google-analytics.com","googletagservices.com","adservice.google","googletag.","pagead2.googlesyndication","facebook.net/tr","connect.facebook.net","fbcdn.net/rsrc","analytics.twitter.com","static.ads-twitter.com","ads.twitter.com","t.co/i/adsct","pixel.advertising.com","hotjar.com","clarity.ms","quantserve.com","scorecardresearch.com","newrelic.com","nr-data.net","segment.io","segment.com","amplitude.com","mixpanel.com","heap.io","fullstory.com","moatads.com","criteo.com","adsrvr.org","rubiconproject.com","pubmatic.com","openx.net","appnexus.com","amazon-adsystem.com","media.net","yieldmo.com","triplelift.com","sharethrough.com","prebid.","smaato.net","indexworm.com","casalemedia.com","outbrain.com","taboola.com","revcontent.com","mgid.com","tawk.to","intercom.io","drift.com","hs-scripts.com","zopim.com","livechatinc.com","userlike.com","onetrust.com","cookielaw.org","cookieinformation.com","trustarc.com","/ads/","/ad/","/banner/","/banners/","/pixel/","/beacon/","/tracking/","/tracker/","/remarketing/","/conversion/","1x1.gif","spacer.gif","blank.gif","transparent.gif"],ae=new Set([".jpg",".jpeg",".png",".webp",".gif",".avif",".svg",".tiff"]),de=new Set([".mp4",".webm",".mov",".avi",".m4v",".ogv",".mkv"]),ce=new Set([".mp3",".wav",".ogg",".aac",".m4a",".flac",".opus"]);function le(e){let t=e.toLowerCase();return se.some(r=>t.includes(r))}function ue(e){return e.startsWith("data:")}function F(e){try{let t=k(new URL(e).pathname).toLowerCase();if(ae.has(t))return"image";if(de.has(t))return"video";if(ce.has(t))return"audio"}catch{}return null}function q(e){let t=e.toLowerCase();return t.startsWith("image/")?"image":t.startsWith("video/")?"video":t.startsWith("audio/")?"audio":null}function me(e,t){if(!e||ue(e))return null;try{return new URL(e,t).href}catch{return null}}function U(e,t){try{let r=new URL(e);return B(r.pathname).replace(/[^a-zA-Z0-9._-]/g,"_").slice(0,80)||`asset-${t}`}catch{return`asset-${t}`}}function O(e,t){let r=e?.replace(/&(?:amp|#0?38|#x26);/gi,"&").replace(/\s+/g," ").trim();return r?r.slice(0,t):null}function ge(e){let t=e.match(/[-_/](\d{2,5})x(\d{2,5})(?=[._/?#-]|$)/i);return t?{width:Number(t[1]),height:Number(t[2])}:null}function pe(e){try{let t=new URL(e);t.hash="",t.pathname=t.pathname.replace(/-(?:\d{2,5}x\d{2,5}|scaled|e\d{6,})(?=\.[a-z0-9]{2,6}$)/i,"");for(let r of["w","width","h","height","resize"])t.searchParams.delete(r);return t.searchParams.sort(),t.href}catch{return e}}function S(e){let t=(e.width??0)*(e.height??0),r=e.discoveryMethods.some(g=>g.includes("lightbox"))?1e12:0,n=/-(?:\d{2,5}x\d{2,5}|scaled|e\d{6,})(?=\.[a-z0-9]{2,6}(?:[?#]|$))/i.test(e.url)?0:1e11;return r+n+t}function A(e,t){let r=e.get(t.url);if(!r){e.set(t.url,t);return}r.discoveryMethods=[...new Set([...r.discoveryMethods,...t.discoveryMethods])],r.altTexts=[...new Set([...r.altTexts,...t.altTexts])],r.contexts=[...new Set([...r.contexts,...t.contexts])],!r.type&&t.type&&(r.type=t.type),(t.width??0)*(t.height??0)>(r.width??0)*(r.height??0)&&(r.width=t.width,r.height=t.height)}function he(e,t){let r=new Map,n=(d,i,o=null,h,l,u)=>{let s=d?me(d.trim().replace(/&/g,"&"),t):null;if(!s||le(s))return;let m=u??ge(s);A(r,{url:s,type:o??F(s),discoveryMethods:[i],altTexts:h?[h]:[],contexts:l?[l]:[],width:m?.width??null,height:m?.height??null})},g=(d,i,o,h)=>{for(let l of(d??"").split(",")){let[u,s]=l.trim().split(/\s+/);if(!u)continue;let m=s?.match(/^(\d+)w$/)?.[1];n(u,i,o,h,null,m?{width:Number(m),height:0}:null)}},a=(d,i="")=>{let o=d.$;for(let l of d.images){let u=l.attributes,s=O(l.alt,500),m=l.width&&l.height?{width:l.width,height:l.height}:null;for(let p of["src","data-src","data-lazy-src","data-original","data-bg","data-background","data-bg-url","data-lazy-bg","data-echo"]){let y=u[p];n(y,`static-${p}${i}`,"image",s,null,m)}g(u.srcset,`static-srcset${i}`,"image",s),g(u["data-srcset"],`static-data-srcset${i}`,"image",s)}o("source").each((l,u)=>{let s=o(u),m=s.attr("type")??"",p=m.startsWith("video/")?"video":m.startsWith("audio/")?"audio":"image";n(s.attr("src"),`static-source${i}`,p),g(s.attr("srcset"),`static-source-srcset${i}`,p)}),o("video,audio").each((l,u)=>{let s=o(u),m=u.tagName.toLowerCase();n(s.attr("src"),`static-${m}${i}`,m),m==="video"&&n(s.attr("poster"),`static-video-poster${i}`,"image")}),o("meta").each((l,u)=>{let s=o(u),m=(s.attr("property")??s.attr("name")??"").toLowerCase();(m==="og:image"||m==="og:image:url"||m==="twitter:image"||m==="twitter:image:src")&&n(s.attr("content"),`static-${m}${i}`,"image")}),o("link[href]").each((l,u)=>{let s=o(u);(s.attr("rel")??"").toLowerCase().split(/\s+/).some(p=>p==="icon"||p==="apple-touch-icon")&&n(s.attr("href"),`static-icon${i}`,"image")}),o("svg image").each((l,u)=>{let s=o(u);n(s.attr("href")??s.attr("xlink:href"),`static-svg-image${i}`,"image")}),o("a[href]").has("img").each((l,u)=>{let s=o(u),m=s.attr("href");if(m&&(/lightbox|gallery|fancybox|glightbox|swipebox|elementor-open-lightbox/i.test(`${s.attr("class")??""} ${s.attr("data-elementor-open-lightbox")??""}`)||/\.(?:avif|gif|jpe?g|png|svg|webp)(?:[?#]|$)/i.test(m))){let p=O(s.find("img").first().attr("alt"),500);n(m,`static-lightbox-target${i}`,"image",p)}});let h=o("style").toArray().map(l=>o(l).text());o("[style]").each((l,u)=>{h.push(o(u).attr("style")??"")});for(let l of h.join(`
+import{a as N}from"./chunk-YQZGZBB4.js";import{b as L,c as P}from"./chunk-HE45FFBU.js";import{h as $,s as x}from"./chunk-63T4W2LX.js";import{createWriteStream as Z,mkdirSync as ee,rmSync as te}from"fs";import{homedir as re}from"os";import{join as I,extname as k,basename as B}from"path";import{pipeline as ne}from"stream/promises";import{Readable as oe,Transform as ie}from"stream";var se=["doubleclick.net","googlesyndication.com","googletagmanager.com","google-analytics.com","googletagservices.com","adservice.google","googletag.","pagead2.googlesyndication","facebook.net/tr","connect.facebook.net","fbcdn.net/rsrc","analytics.twitter.com","static.ads-twitter.com","ads.twitter.com","t.co/i/adsct","pixel.advertising.com","hotjar.com","clarity.ms","quantserve.com","scorecardresearch.com","newrelic.com","nr-data.net","segment.io","segment.com","amplitude.com","mixpanel.com","heap.io","fullstory.com","moatads.com","criteo.com","adsrvr.org","rubiconproject.com","pubmatic.com","openx.net","appnexus.com","amazon-adsystem.com","media.net","yieldmo.com","triplelift.com","sharethrough.com","prebid.","smaato.net","indexworm.com","casalemedia.com","outbrain.com","taboola.com","revcontent.com","mgid.com","tawk.to","intercom.io","drift.com","hs-scripts.com","zopim.com","livechatinc.com","userlike.com","onetrust.com","cookielaw.org","cookieinformation.com","trustarc.com","/ads/","/ad/","/banner/","/banners/","/pixel/","/beacon/","/tracking/","/tracker/","/remarketing/","/conversion/","1x1.gif","spacer.gif","blank.gif","transparent.gif"],ae=new Set([".jpg",".jpeg",".png",".webp",".gif",".avif",".svg",".tiff"]),de=new Set([".mp4",".webm",".mov",".avi",".m4v",".ogv",".mkv"]),ce=new Set([".mp3",".wav",".ogg",".aac",".m4a",".flac",".opus"]);function le(e){let t=e.toLowerCase();return se.some(r=>t.includes(r))}function ue(e){return e.startsWith("data:")}function F(e){try{let t=k(new URL(e).pathname).toLowerCase();if(ae.has(t))return"image";if(de.has(t))return"video";if(ce.has(t))return"audio"}catch{}return null}function q(e){let t=e.toLowerCase();return t.startsWith("image/")?"image":t.startsWith("video/")?"video":t.startsWith("audio/")?"audio":null}function me(e,t){if(!e||ue(e))return null;try{return new URL(e,t).href}catch{return null}}function U(e,t){try{let r=new URL(e);return B(r.pathname).replace(/[^a-zA-Z0-9._-]/g,"_").slice(0,80)||`asset-${t}`}catch{return`asset-${t}`}}function O(e,t){let r=e?.replace(/&(?:amp|#0?38|#x26);/gi,"&").replace(/\s+/g," ").trim();return r?r.slice(0,t):null}function ge(e){let t=e.match(/[-_/](\d{2,5})x(\d{2,5})(?=[._/?#-]|$)/i);return t?{width:Number(t[1]),height:Number(t[2])}:null}function pe(e){try{let t=new URL(e);t.hash="",t.pathname=t.pathname.replace(/-(?:\d{2,5}x\d{2,5}|scaled|e\d{6,})(?=\.[a-z0-9]{2,6}$)/i,"");for(let r of["w","width","h","height","resize"])t.searchParams.delete(r);return t.searchParams.sort(),t.href}catch{return e}}function S(e){let t=(e.width??0)*(e.height??0),r=e.discoveryMethods.some(g=>g.includes("lightbox"))?1e12:0,n=/-(?:\d{2,5}x\d{2,5}|scaled|e\d{6,})(?=\.[a-z0-9]{2,6}(?:[?#]|$))/i.test(e.url)?0:1e11;return r+n+t}function A(e,t){let r=e.get(t.url);if(!r){e.set(t.url,t);return}r.discoveryMethods=[...new Set([...r.discoveryMethods,...t.discoveryMethods])],r.altTexts=[...new Set([...r.altTexts,...t.altTexts])],r.contexts=[...new Set([...r.contexts,...t.contexts])],!r.type&&t.type&&(r.type=t.type),(t.width??0)*(t.height??0)>(r.width??0)*(r.height??0)&&(r.width=t.width,r.height=t.height)}function he(e,t){let r=new Map,n=(d,i,o=null,h,l,u)=>{let s=d?me(d.trim().replace(/&/g,"&"),t):null;if(!s||le(s))return;let m=u??ge(s);A(r,{url:s,type:o??F(s),discoveryMethods:[i],altTexts:h?[h]:[],contexts:l?[l]:[],width:m?.width??null,height:m?.height??null})},g=(d,i,o,h)=>{for(let l of(d??"").split(",")){let[u,s]=l.trim().split(/\s+/);if(!u)continue;let m=s?.match(/^(\d+)w$/)?.[1];n(u,i,o,h,null,m?{width:Number(m),height:0}:null)}},a=(d,i="")=>{let o=d.$;for(let l of d.images){let u=l.attributes,s=O(l.alt,500),m=l.width&&l.height?{width:l.width,height:l.height}:null;for(let p of["src","data-src","data-lazy-src","data-original","data-bg","data-background","data-bg-url","data-lazy-bg","data-echo"]){let y=u[p];n(y,`static-${p}${i}`,"image",s,null,m)}g(u.srcset,`static-srcset${i}`,"image",s),g(u["data-srcset"],`static-data-srcset${i}`,"image",s)}o("source").each((l,u)=>{let s=o(u),m=s.attr("type")??"",p=m.startsWith("video/")?"video":m.startsWith("audio/")?"audio":"image";n(s.attr("src"),`static-source${i}`,p),g(s.attr("srcset"),`static-source-srcset${i}`,p)}),o("video,audio").each((l,u)=>{let s=o(u),m=u.tagName.toLowerCase();n(s.attr("src"),`static-${m}${i}`,m),m==="video"&&n(s.attr("poster"),`static-video-poster${i}`,"image")}),o("meta").each((l,u)=>{let s=o(u),m=(s.attr("property")??s.attr("name")??"").toLowerCase();(m==="og:image"||m==="og:image:url"||m==="twitter:image"||m==="twitter:image:src")&&n(s.attr("content"),`static-${m}${i}`,"image")}),o("link[href]").each((l,u)=>{let s=o(u);(s.attr("rel")??"").toLowerCase().split(/\s+/).some(p=>p==="icon"||p==="apple-touch-icon")&&n(s.attr("href"),`static-icon${i}`,"image")}),o("svg image").each((l,u)=>{let s=o(u);n(s.attr("href")??s.attr("xlink:href"),`static-svg-image${i}`,"image")}),o("a[href]").has("img").each((l,u)=>{let s=o(u),m=s.attr("href");if(m&&(/lightbox|gallery|fancybox|glightbox|swipebox|elementor-open-lightbox/i.test(`${s.attr("class")??""} ${s.attr("data-elementor-open-lightbox")??""}`)||/\.(?:avif|gif|jpe?g|png|svg|webp)(?:[?#]|$)/i.test(m))){let p=O(s.find("img").first().attr("alt"),500);n(m,`static-lightbox-target${i}`,"image",p)}});let h=o("style").toArray().map(l=>o(l).text());o("[style]").each((l,u)=>{h.push(o(u).attr("style")??"")});for(let l of h.join(`
 `).matchAll(/(?:background(?:-image)?\s*:\s*)?url\(\s*(?:["']|&quot;|&#0?34;|&apos;|&#0?39;)?([^"')&]+)(?:["']|&quot;|&#0?34;|&apos;|&#0?39;)?\s*\)/gi))n(l[1],`static-css-url${i}`,"image");for(let l of d.sourceHtml.matchAll(/["'](https?:\/\/[^"'\s<>]+\.(?:avif|gif|jpe?g|png|svg|webp|mp4|webm|mp3|ogg)(?:\?[^"'\s<>]*)?)["']/gi))n(l[1],`static-bare-url${i}`)},c=typeof e=="string"?P(e):e;return a(c),c.$("script").each((d,i)=>{let o=c.$(i).text(),h=o.replace(/\\u003[cC]/g,"<").replace(/\\u003[eE]/g,">").replace(/\\u0026/gi,"&").replace(/\\["']/g,l=>l.slice(1)).replace(/\\\//g,"/");h!==o&&/<(?:img|source|video|audio)\b/i.test(h)&&a(P(h),"-json-unescaped")}),[...r.values()]}function fe(e,t,r=100){let n=new Map;for(let a of e)A(n,a);for(let a of t?.assets??[])A(n,{...a,type:a.type});let g=new Map;for(let a of n.values()){let c=pe(a.url),d=g.get(c)??[];d.push(a),g.set(c,d)}return[...g.values()].map(a=>{let c=[...a].sort((i,o)=>S(o)-S(i)||i.url.localeCompare(o.url)),d={...c[0]};return d.discoveryMethods=[...new Set(a.flatMap(i=>i.discoveryMethods))],d.altTexts=[...new Set(a.flatMap(i=>i.altTexts))],d.contexts=[...new Set(a.flatMap(i=>i.contexts))],{...d,variants:c.slice(1).map(i=>i.url)}}).sort((a,c)=>S(c)-S(a)||a.url.localeCompare(c.url)).slice(0,Math.max(1,r))}async function ye(e,t,r,n={}){let g=Math.max(1,Math.min(n.maxBytes??52428800,104857600)),a=e,c=null;for(let p=0;p<=5;p++){let y=await L(a,{field:"media URL"});if(y.error||!y.parsed)throw new Error(y.error??"Media URL was rejected");if(c=await fetch(y.parsed.href,{signal:AbortSignal.timeout(15e3),redirect:"manual"}),c.status>=300&&c.status<400){let w=c.headers.get("location");if(!w)throw new Error(`HTTP ${c.status} redirect did not include Location`);a=new URL(w,y.parsed.href).href,c=null;continue}break}if(!c)throw new Error("Media download exceeded five redirects");if(!c.ok)throw new Error(`HTTP ${c.status}`);if(!c.body)throw new Error("Empty response body");let d=c.headers.get("content-type")?.split(";")[0].trim()??null;if(n.expectedType&&(!d||q(d)!==n.expectedType))throw new Error(`Expected ${n.expectedType} content but received ${d??"no content-type"}`);let i=Number(c.headers.get("content-length")??0);if(Number.isFinite(i)&&i>g)throw new Error(`Media asset exceeds ${g} byte limit`);let o=I(t,r);if(d&&!k(r)){let y={"image/jpeg":".jpg","image/png":".png","image/webp":".webp","image/gif":".gif","image/svg+xml":".svg","image/avif":".avif","video/mp4":".mp4","video/webm":".webm","audio/mpeg":".mp3","audio/ogg":".ogg","audio/wav":".wav"}[d];y&&(o=o+y)}let h=Z(o);await new Promise((p,y)=>{h.once("open",()=>p()),h.once("error",y)});let l=0,u=new ie({transform(p,y,w){let _=p.length;if(l+_>g){w(new Error(`Media asset exceeds ${g} byte limit`));return}if(n.consumeBytes&&!n.consumeBytes(_)){w(new Error("Media export total byte limit exceeded"));return}l+=_,w(null,p)}});try{await ne(oe.fromWeb(c.body),u,h)}catch(p){throw h.destroy(),te(o,{force:!0}),p}let{statSync:s}=await import("fs"),m=s(o).size;return{savedPath:o,sizeBytes:m,mimeType:d}}async function Le(e,t,r={}){let n=r.types??["image","video","audio"],g=new Set(n),a=he(e,t),c=Math.max(1,Math.min(r.maxAssets??100,250)),d=fe(a,r.rendered,2e3),i=d.slice(0,c),o=new Set([...a.map(f=>f.url),...(r.rendered?.assets??[]).map(f=>f.url)]).size,h=i.flatMap(f=>{let b=f.type??F(f.url);return b&&g.has(b)?[{...f,type:b}]:[]}),l=o-h.length,u=d.length>c,s=r.rendered?.exhausted===!0&&!u,m=u?"asset_limit":r.rendered?.stopReason??"render_unavailable",p=u?[`Media discovery found ${d.length} responsive families; only the requested ${c} records were retained.`]:r.rendered?r.rendered.exhausted?[]:[`Rendered media discovery stopped with ${r.rendered.stopReason}; the inventory may be incomplete.`]:["Rendered media discovery was unavailable; this manifest contains static-source evidence only."],y={pageUrl:t,staticFound:a.length,renderedFound:r.rendered?.assets.length??0,filteredCount:l,totalFound:o,completeness:s?"complete":"partial",exhausted:s,stopReason:m,scrollRounds:r.rendered?.scrollRounds??0,warnings:p,artifact:null};if(r.outputDir===null)return{...y,outputDir:null,assets:h.map((f,b)=>({...f,mimeType:null,filename:U(f.url,b),savedPath:null,sizeBytes:null,sha256:null,downloadStatus:"not_attempted",downloadError:null,inlinePreview:null}))};let w=(()=>{try{return new URL(t).hostname.replace(/^www\./,"")}catch{return"unknown"}})(),_=new Date().toISOString().replace(/[:.]/g,"-").slice(0,19),C=r.outputDir??I(re(),"Downloads","mcp-scraper","media",`${_}-${w}`);ee(C,{recursive:!0});let T=[];return await Promise.allSettled(h.map(async(f,b)=>{let R=U(f.url,b);try{let{savedPath:v,sizeBytes:Q,mimeType:D}=await ye(f.url,C,R),G=D?q(D)??f.type:f.type;T.push({...f,type:G,mimeType:D,filename:B(v),savedPath:v,sizeBytes:Q,sha256:null,downloadStatus:"downloaded",downloadError:null,inlinePreview:null})}catch(v){T.push({...f,mimeType:null,filename:R,savedPath:null,sizeBytes:null,sha256:null,downloadStatus:"failed",downloadError:v instanceof Error?v.message:"media_download_failed",inlinePreview:null})}})),T.sort((f,b)=>f.savedPath&&!b.savedPath?-1:!f.savedPath&&b.savedPath?1:f.url.localeCompare(b.url)),{...y,outputDir:C,assets:T}}var be=new Set(["browser_navigation_blocked","browser_navigation_failed","browser_timeout","browser_session_failed","browser_result_missing"]);function we(e){return e instanceof Error?e.code:void 0}function _e(e){return e instanceof Error?e.message:String(e)}function H(e){let t=e?.match(/^browser_http_(\d+)$/);return t?Number(t[1]):null}function ve(e){return e===404?"page_not_found":e===403?"page_forbidden":e===429?"page_rate_limited":e>=500&&e<=599?"page_server_error":"page_http_error"}function xe(e){let t=_e(e),r=we(e);if(N(e,t))return{errorCode:"vendor_unavailable",httpStatus:null,retryable:!0};let n=H(r)??H(t.match(/^(browser_http_\d+)/)?.[1]);return n!=null?{errorCode:ve(n),httpStatus:n,retryable:n===429||n>=500}:r==="browser_challenge_unresolved"||/^browser_challenge_unresolved/.test(t)?{errorCode:"bot_check_unresolved",httpStatus:null,retryable:!0}:r==="browser_response_too_large"||/exceeds \d+ byte limit/i.test(t)?{errorCode:"page_too_large",httpStatus:null,retryable:!1}:/browser (?:has been )?closed|context closed|target page, context or browser has been closed|session closed/i.test(t)?{errorCode:"browser_session_interrupted",httpStatus:null,retryable:!0}:be.has(r??"")||/timeout|navigation_failed|ERR_/i.test(t)?{errorCode:"page_unreachable",httpStatus:null,retryable:!0}:{errorCode:"extraction_failed",httpStatus:null,retryable:!1}}function j(e,t){switch(e){case"page_not_found":return`The page could not be found (HTTP ${t??404}). It may have been moved or deleted.`;case"page_forbidden":return`The site refused the request (HTTP ${t??403}).`;case"page_rate_limited":return"The site is rate-limiting requests right now (HTTP 429). Retrying after a short wait usually works.";case"page_server_error":return`The page's own server returned an error (HTTP ${t}). This is on the target site's side, not something a retry here can fix.`;case"page_http_error":return`The page returned an unexpected HTTP ${t} response.`;case"bot_check_unresolved":return"This site has automated-traffic protection (a bot/CAPTCHA check) that could not be resolved in time. Some sites are simply not extractable this way.";case"page_too_large":return"The page is larger than we can safely process.";case"page_unreachable":return"The page did not respond in time, or the connection could not be completed.";case"browser_session_interrupted":return"The browser session closed before extraction completed. A retry uses a fresh browser session.";case"vendor_unavailable":return"MCP Scraper's internal browser services require MCP Scraper team attention. Servers or IPs are down until this is fixed \u2014 this is not caused by your request. Please retry in a few minutes.";case"extraction_failed":return"The page could not be extracted. Please retry, or contact support if this persists for the same URL."}}function Ie(e,t={}){let r=j(e.errorCode,e.httpStatus);return{...$({errorCode:e.errorCode,retryable:e.retryable,chargeStatus:t.chargeStatus,details:{...t.details??{},...e.httpStatus!=null?{http_status:e.httpStatus}:{}}}),message:r,error:r}}function ke(e){return e.httpStatus===404?404:e.httpStatus===403?403:e.httpStatus===429?429:e.httpStatus!=null&&e.httpStatus>=500&&e.httpStatus<=599?500:e.httpStatus!=null?502:e.errorCode==="vendor_unavailable"?503:502}var Ee="wayback_capture_missing";function Be(e,t){if(e==null)return{failureCode:null,failureReason:null};if(e===Ee)return{failureCode:e,failureReason:t??null};let r=new Error(t??e);r.code=e;let n=xe(r);return{failureCode:n.errorCode,failureReason:j(n.errorCode,n.httpStatus)}}import{neon as Te}from"@neondatabase/serverless";var Se="jina",M=null,z=!1;function W(){return(process.env.JINA_EMBED_MODEL??"jina-embeddings-v5-omni-small").trim()}function Y(){return Number((process.env.JINA_EMBED_DIM??"1024").trim())}function J(){return!!(process.env.JINA_API_KEY?.trim()&&process.env.MEMORY_DATABASE_URL?.trim())}function E(){if(M)return M;let e=process.env.MEMORY_DATABASE_URL?.trim();if(!e)throw new Error("MEMORY_DATABASE_URL is not set; Commons semantic search needs the shared Postgres.");return M=Te(e),M}async function K(){if(z)return;let e=Y();await E().query("CREATE EXTENSION IF NOT EXISTS vector"),await E().query(`
     CREATE TABLE IF NOT EXISTS commons_index_vectors (
       document_id   TEXT PRIMARY KEY,

package/dist/chunk-5H3MCTTP.js

@@ -0,0 +1 @@
+var o="0.89.1";export{o as a};