mcp-scraper 0.34.1 → 0.36.0

package/docs/specs/query-fanout-transport-contract-fix.md

@@ -1,6 +1,6 @@
 # Query fan-out transport contract fix
 
-- **Status:** Implemented; pending deployed live proof
+- **Status:** Released and verified in production
 - **Baseline:** `origin/main` at `e2c90f8db7c22fa2aa48fc62cef10e2e0c799e15` (2026-07-25)
 - **Decision:** [Keep hosted fan-out results inline](../decisions/2026-07-25-hosted-fanout-results-stay-inline.md)
 - **Owner:** MCP Scraper browser-agent surface
@@ -32,6 +32,14 @@ Focused tests must prove hosted calls force upstream `export: false` and return
 
 Live release acceptance requires an explicitly authorized new web-search prompt: hosted OAuth with `export: true` must return a populated inline result with `exports: null`; stdio with the same option must additionally produce readable local exports.
 
+## Release evidence
+
+- PR [#64](https://github.com/VilovietaSEO/mcp-scraper/pull/64) merged as `34a5eefa69b1fcaa2dec1dd645ac0dfc55cf4035`.
+- npm, annotated tag, and GitHub/MCPB release published as `0.34.1` / `v0.34.1`.
+- Vercel production deployment `dpl_CFCSkpAywT6PyqRe8LPDaKvicGrp` reached `READY` and serves `mcpscraper.dev`.
+- Hosted Streamable HTTP with OAuth exposed 167 tools and returned a successful fan-out with 21 researched sources, complete inline data, `exports: null`, and no export error.
+- The exact published npm `0.34.1` stdio package exposed 167 tools, returned a successful fan-out with 30 researched sources, and produced nine readable local export files with no export error.
+
 ## Out of scope
 
 Public/shared download URLs, capture-parser changes, profile authentication, billing, timeout changes, and recovery of fan-out from pre-capture prompts.

package/docs/specs/unified-credit-and-scheduled-execution-billing-spec.md

@@ -1,36 +1,39 @@
 # Unified Credits, Integrations, and Scheduled Execution Billing
 
-- Status: Released v1 contract plus explicitly deferred hardening roadmap
-- Spec date: 2026-07-14
+- Status: Released scheduled-execution contract plus approved connected-account v2 contract
+- Spec date: 2026-07-15
 - Primary repos: mcp-scraper, mcp-scraper-scheduler, mcpscraper-web, mcpscraper-sdk
 - Commercial decision: Scheduled Actions is a product capability, not a separate subscription
+- Connected-account decision: each active Nango connection is a $3/month add-on; usage is metered in Credits
 - Accounting decision: MCP Scraper's main credit ledger is the only debit authority
 
 ## Executive decision
 
 MCP Scraper will use one visible usage currency: Credits.
 
-Scheduled Actions uses the same Credit wallet as the rest of MCP Scraper. It is no longer sold as a separate $10-per-month product with a 1,000-run quota. Integrations themselves are included with paid plans: connecting, reconnecting, direct reads, enabled actions, supported exports, and connection-to-Memory snapshots do not currently create a separate integration-operation debit.
+Scheduled Actions uses the same Credit wallet as the rest of MCP Scraper. It is no longer sold as a separate $10-per-month product with a 1,000-run quota. Connected accounts are commercial add-ons: each active Nango connection costs $3 per month, and provider usage is settled from the shared Credit wallet using authoritative receipts.
 
 The replacement contract is:
 
-1. Every paid MCP Scraper plan includes access to integrations and scheduling.
-2. OAuth connect and reconnect cost 0 Credits and there is no per-connection subscription or active-connection minimum.
+1. Every paid MCP Scraper plan includes access to scheduling and the ability to add connected accounts.
+2. OAuth connect and reconnect cost 0 Credits. Each active Nango connection is a separate $3-per-month recurring add-on, prorated by the subscription system.
 3. Each scheduled occurrence that passes the balance check and begins execution costs 75 Credits exactly once.
 4. A deterministic `connection_sync` occurrence has no model charge beyond that 75-Credit base.
 5. An agent-mode occurrence additionally settles OpenRouter's actual reported model cost, multiplied by 1.5, then converted to Credits.
 6. Token prices may be shown as current per-million-token estimates in Credits, but OpenRouter's reported cost is the settlement truth.
 7. Separately metered scraper, browser, transcription, video-analysis, and similar tools keep their published rate-card prices when a schedule calls them.
-8. The proposed 5/10/15/25-Credit connected-operation tariff, per-record/byte tariff, and 20,000-Credit active-connection minimum are deferred and are not enabled.
+8. Connected usage costs 2 Credits per Nango function run, 2 Credits per Nango Proxy request, and 5 Credits per compute second, prorated from aggregate milliseconds and rounded once up to integer mc.
+9. The older 5/10/15/25-Credit semantic read/write tariff, per-record/byte tariff, and 20,000-Credit active-connection minimum remain retired and are not enabled.
 
-This design encourages usage: paid users can immediately connect and use accounts without per-operation friction, while recurring automation and model work draw from the shared Credit balance. There is no artificial 1,000-run cliff or add-on product to understand.
+This design separates the predictable credential-carrying cost from variable work. The $3 recurring line item pays for keeping a Nango connection active; function, Proxy, and compute activity draw from the shared Credit balance. There is no artificial 1,000-run cliff or separate scheduler product.
 
 ### Released v1 implementation boundary
 
 - Authorization currently performs an idempotent wallet balance check, not a hard balance reservation.
 - The server-side default maximum is 5,000 Credits per run and is environment-configurable; it is not yet editable in the customer UI.
 - A missing authoritative model cost moves the schedule to `cost_pending` and pauses it. Automated recovery/reconciliation is still a hardening item.
-- Connected-operation receipts, user-editable monthly caps, auto-refill, and active-connection minimums remain future work unless a later commercial decision explicitly enables them.
+- The main API accepts idempotent connected-usage receipts and is the settlement authority. The scheduler must preflight at least the fixed 2-Credit function unit, and a prior `cost_pending` connected receipt blocks new work until recovered.
+- User-editable monthly caps, auto-refill, hard reservations, and automated Nango-to-ledger reconciliation remain hardening work.
 
 ## What was found before the v1 cutover
 
@@ -66,25 +69,25 @@ Do not assume existing scheduled-agent charges equal a clean 1.5 markup. Before
 
 ## Canonical Build Spec
 
-- **Request:** Replace separate Scheduled Actions billing with the shared Credit wallet; charge one 75-Credit occurrence base and correctly settle all-step OpenRouter cost at 1.5; keep direct integration use included.
+- **Request:** Keep Scheduled Actions on the shared Credit wallet; charge one 75-Credit occurrence base and correctly settle all-step OpenRouter cost at 1.5; add a $3 active-Nango-connection recurring line item plus receipt-based function, Proxy, and millisecond compute usage.
 - **Lane:** Mastra Agentic Development plus MCP/API product billing.
 - **Capability:** Unified Metered Execution.
 - **Audience:** MCP Scraper customers, scheduled-agent users, MCP clients, SDK users, operators, finance, and support.
-- **User-Facing Outcome:** A user connects a service, uses it directly without a separate operation fee, optionally moves a bounded result into Memory, and pays 75 Credits plus applicable model/tool usage when recurring work executes.
+- **User-Facing Outcome:** A user pays $3/month for each active Nango account, sees connected usage itemized in Credits, optionally schedules work, and receives idempotent receipts without duplicate debits on retries.
 - **Target Runtime:** Main MCP Scraper API and credit ledger; Vercel-hosted Mastra scheduler; Inngest workers; Nango connections and Proxy/functions; private Blob artifacts; public MCP; npm/SDK clients.
 - **Inputs:** Authenticated identity, connection ID, tool or sync request, scheduled occurrence, model selection, per-run and monthly Credit limits, actual provider results, OpenRouter usage metadata, bytes and records delivered.
 - **Agents:** Existing Scheduled Action Agent for interpretive work. Deterministic connection_sync workers remain non-agentic. Both use the same billing context.
 - **Workflows:** Connect; direct read/write; bulk export; memory import; schedule estimate; occurrence claim; Credit authorization; execution; incremental settlement; final settlement; receipt; retry/reconciliation.
 - **Tools:** Existing generic connected-service tools, provider-native tools, schedule tools, bulk export/import tools, credits_info, a new schedule estimator, and a shared internal billing adapter.
 - **MCP:** Credit behavior is transport-independent. Tool descriptions state charge categories and estimates, results include receipt summaries, and exact provider tools remain protected by tenant ownership, action switches, and grants.
-- **Signals:** Billing authorization created; low balance; budget threshold; cost pending; settlement complete; and insufficient Credits. Auto-refill and connection-minimum signals belong only to the deferred roadmap.
+- **Signals:** Billing authorization created; connected-use preflight; low balance; budget threshold; cost pending; settlement complete; and insufficient Credits.
 - **Background Tasks:** Usage aggregation, connection-minimum settlement, pending-cost reconciliation, low-balance notifications, automatic refill, legacy subscription migration, receipt retention, and rate-snapshot refresh.
 - **Guardrails:** Integer microcredit arithmetic, idempotent charges, one debit authority, no model-visible billing credentials, fail closed before externally costly work, action approval independent of price, no prompt or provider credential in billing events.
 - **Observability:** Itemized receipt per run; raw vendor cost; multiplier; model/provider; tokens; bytes; changed records; connection and schedule IDs; idempotency keys; authorization, settlement, and refund state.
 - **Enforcement:** A call cannot bypass the billing adapter by using Mastra, the generic MCP bridge, direct REST, SDK, Nango, or a background job. Duplicate occurrence or HTTP retry cannot debit twice.
 - **Persistence:** The main MCP Scraper/Turso credit ledger owns balances and debits. Scheduler Postgres owns schedule state, occurrence state, and a mirrored receipt reference, not the authoritative wallet.
 - **Deployment:** Main API/ledger first, then scheduler and connection adapters, then public MCP, frontend, SDK/npm/MCPB, generated docs, Stripe migration, and Vercel production.
-- **Done Contract:** A production scheduled run performs one balance authorization, charges its 75-Credit base once, aggregates every OpenRouter model step once, and exposes the same pricing contract through the UI, MCP, API, and SDK. Hard reservations and full receipt UX remain the next hardening phase.
+- **Done Contract:** A production scheduled run charges its 75-Credit base once and aggregates every OpenRouter model step once. Every Nango call is preflighted, reports function/Proxy/compute usage under the active rate revision, and settles once in the main ledger. The UI, MCP, API, SDK, and recurring connection quantity publish the same contract.
 - **Receipt Requirements:** Code and deployment SHAs; rate-table revision; Stripe migration result; test output; production Vercel IDs; npm/MCPB versions; sample agent and deterministic-sync billing receipts; reconciliation report; public pricing and tool-description parity.
 
 ## Product contract in plain language
@@ -106,7 +109,7 @@ A scheduled occurrence pays:
     + any separately metered scraper, browser, transcription,
       video-analysis, or other published rate-card tools it calls
 
-Direct connected-service reads, enabled actions, supported exports, and bounded Memory snapshots do not add a separate integration-operation line item today.
+Direct connected-service reads, enabled actions, supported exports, and bounded Memory snapshots report the Nango resources they actually consume: function runs, Proxy requests, and function compute milliseconds. The tool's semantic label does not change those infrastructure rates.
 
 Internal pagination, retries, leases, and orchestration steps are not separate “runs.” A retry of the same occurrence cannot pay the fixed charge again.
 
@@ -182,21 +185,26 @@ Released v1 rates:
 | Billing class | Charge |
 |---|---:|
 | OAuth connect, reconnect, or consent callback | Free |
-| Direct approved read | Included |
-| Direct enabled action | Included |
-| Supported connected-service export | Included |
-| Bounded connection-to-Memory snapshot | Included |
-| Active connection minimum | None |
+| Active Nango connection | $3 per month, recurring and prorated |
+| Nango function run | 2 Credits per run |
+| Nango Proxy request | 2 Credits per request |
+| Nango function compute | 5 Credits per second, prorated from aggregate milliseconds |
 
 Action safety remains independent of price: ownership, live inventory, source-controlled policy, the account action switch, and exact scheduled grants still apply.
 
+The active rate revision is `2026-07-15.connected-usage-v1`. Compute settlement uses:
+
+    compute mc = ceil(computeMilliseconds * 500 mc / 1000)
+
+The scheduler aggregates raw milliseconds for one receipt before applying that ceiling. It does not round each span or function up to a whole second. Function, Proxy, and compute line items are additive; the recurring $3 connection charge is not converted into Credits.
+
 ### Deferred connected-operation pricing options (not enabled)
 
 Earlier design work proposed 5/10/15/25-Credit read/write classes, a 25-Credit bulk base, per-record charges, and per-MiB charges. Those values are retained only as research for a later commercial decision. They are not present in the production registry and must not appear as current pricing in UI, MCP descriptions, SDKs, or support answers.
 
-### Deferred active connection minimum (not enabled)
+### Retired active connection minimum proposal
 
-An earlier proposal described a 20,000-Credit minimum per active connection per 30-day billing period. The released product has no active-connection minimum, no first-connection waiver clock, and no connection billing-pause flow. Enabling any such minimum requires a new explicit commercial decision, customer notice, and a separate release.
+An earlier proposal described a 20,000-Credit wallet minimum per active connection per 30-day billing period. It is superseded by the flat $3/month recurring add-on and must not be presented as current pricing.
 
 The remainder of this subsection records the deferred proposal for future evaluation only:
 
@@ -698,7 +706,7 @@ The deterministic worker:
 - claims the same occurrence key;
 - authorizes its budget;
 - charges the 75-Credit occurrence base once;
-- does not add record, byte, or connected-operation charges in released v1;
+- settles observed Nango function runs, Proxy requests, and aggregate compute milliseconds under the active connected-usage rate revision;
 - checkpoints before settlement so retry behavior is deterministic.
 
 The worker never invokes a model merely to paginate or transfer content.
@@ -711,9 +719,9 @@ Remove the Scheduled Actions add-on card.
 
 Under every paid plan:
 
-> Integrations and scheduled work included. Usage draws from your shared Credit balance.
+> Integration access and scheduled work included. Each active Nango account is $3/month; connected usage draws from your shared Credit balance.
 
-Show the 75-Credit scheduled occurrence, the “actual OpenRouter cost plus 50%” disclosure, and the fact that direct integration use is included. Do not show the deferred active-connection minimum or per-operation tariff as current pricing.
+Show the 75-Credit scheduled occurrence, the “actual OpenRouter cost plus 50%” disclosure, the $3 active-account recurring fee, and the 2/2/5 connected-usage rates. Do not show the retired semantic read/write tariff or active-connection Credit minimum as current pricing.
 
 ### Integrations page
 
@@ -940,7 +948,7 @@ Receipts must be sanitized. Do not include prompts, connection tokens, API keys,
 - Separate scheduling entitlement or 1,000-run quota still enforced on any surface.
 - Frontend, hosted MCP, SDK/npm, MCPB, or public docs showing different rates.
 - A fixed token rate described as authoritative rather than an estimate.
-- Any connected-operation tariff or active-connection minimum exposed without a new approved commercial release.
+- Any surface that omits or contradicts the approved $3 plus 2/2/5 connected-account contract.
 - Commercial migration without a reconciliation and compensating-credit policy.
 
 ## Decisions locked by this spec
@@ -951,7 +959,8 @@ Receipts must be sanitized. Do not include prompts, connection tokens, API keys,
 - scheduled_agent_llm uses a 1.5 multiplier.
 - Model token tables are estimates.
 - The fixed scheduled-occurrence charge is 75 Credits.
-- Direct integration use is included; the proposed 5/10/15/25 tariffs and 20,000-Credit active-connection minimum are not enabled.
+- Active Nango connections cost $3/month; usage costs 2 Credits/function execution, 2 Credits/Proxy request, and 5 Credits/compute-second measured from milliseconds.
+- The proposed 5/10/15/25 semantic tariffs and 20,000-Credit active-connection minimum are not enabled.
 - OAuth is free.
 - Main MCP Scraper owns the only mutable Credit ledger.
 - Every product and execution surface publishes the same price contract; shared itemized receipt UX remains hardening work.
@@ -959,8 +968,8 @@ Receipts must be sanitized. Do not include prompts, connection tokens, API keys,
 
 ## Decisions that may change only through a rate revision
 
-- Exact per-read, per-write, per-record, and per-MiB rates.
-- Connection-minimum amount and waiver duration.
+- Exact semantic per-read, per-write, per-record, and per-MiB rates beyond the current Nango resource meter.
+- Connected-account monthly price and function, Proxy, or compute rates.
 - Pay-as-you-go Credit grant per dollar.
 - Default per-run and monthly caps.
 - Existing non-scheduler AI multipliers.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-scraper",
-  "version": "0.34.1",
+  "version": "0.36.0",
   "description": "MCP server for MCP Scraper web intelligence tools",
   "type": "module",
   "main": "./dist/index.cjs",
@@ -35,7 +35,7 @@
     "build:og-pages": "node scripts/build-og-pages.mjs",
     "build:md": "node scripts/build-md-mirror.mjs",
     "build:mcpb": "npm run build && node scripts/build-mcpb.mjs",
-    "build:rates": "tsx scripts/gen-rates.ts",
+    "build:rates": "node --import tsx scripts/gen-rates.ts",
     "build:ui": "npm run build:rates && esbuild public/app.jsx --bundle --loader:.jsx=jsx --outfile=public/app.js --target=es2020",
     "build:blog": "node scripts/run-ts-sandbox-safe.mjs scripts/build-blog.ts",
     "mcp:manifest": "node scripts/run-ts-sandbox-safe.mjs scripts/generate-mcp-tool-manifest.ts",
@@ -52,6 +52,8 @@
     "test:kernel": "vitest run tests/integration/kernel.smoke.test.ts",
     "sync:env": "bash scripts/push-env.sh",
     "sync:env:apply": "bash scripts/push-env.sh --apply",
+    "locations:publish-geonames": "node --import tsx scripts/publish-geonames-location-data.ts",
+    "locations:sync": "node --import tsx scripts/sync-hosted-location-data.ts",
     "integrations:migrate-scheduler": "node --import tsx scripts/migrate-scheduler-integrations.ts",
     "test:webhook": "node scripts/test-webhook.mjs",
     "test:e2e": "playwright test",
@@ -76,6 +78,7 @@
     "diff": "^9.0.0",
     "hono": "^4.12.18",
     "inngest": "^4.4.0",
+    "ipaddr.js": "^1.9.1",
     "jose": "^6.2.3",
     "mcpscraper-memory-tools": "^1.9.0",
     "p-limit": "^7.3.0",
@@ -91,9 +94,6 @@
     "youtube-transcript": "^1.3.1",
     "zod": "^3.23.0"
   },
-  "optionalDependencies": {
-    "@resvg/resvg-js-linux-arm64-gnu": "2.6.2"
-  },
   "devDependencies": {
     "@onkernel/cli": "^0.19.2",
     "@playwright/test": "^1.60.0",
@@ -105,6 +105,7 @@
     "better-sqlite3": "^12.9.0",
     "bgutils-js": "^3.2.0",
     "dotenv": "^17.4.2",
+    "fflate": "^0.7.4",
     "jsdom": "^29.1.1",
     "tsup": "^8.0.0",
     "tsx": "^4.0.0",

package/dist/chunk-5PZ6N2QM.js

@@ -1,130 +0,0 @@
-import {
-  LedgerOperation
-} from "./chunk-BWXLTWF7.js";
-import {
-  sanitizeVendorName
-} from "./chunk-M2S27J6Z.js";
-import {
-  getDb
-} from "./chunk-62DQAWPF.js";
-
-// src/api/site-extract-repository.ts
-function rowToJob(r) {
-  return {
-    id: String(r.id),
-    userId: r.user_id != null ? Number(r.user_id) : null,
-    status: r.status != null ? String(r.status) : "pending",
-    startUrl: String(r.start_url ?? ""),
-    options: r.options ? JSON.parse(String(r.options)) : {},
-    totalUrls: Number(r.total_urls ?? 0),
-    doneUrls: Number(r.done_urls ?? 0),
-    artifacts: r.artifacts ? JSON.parse(String(r.artifacts)) : null,
-    error: r.error != null ? String(r.error) : null,
-    billedMc: r.billed_mc != null ? Number(r.billed_mc) : null,
-    createdAt: String(r.created_at ?? ""),
-    updatedAt: String(r.updated_at ?? "")
-  };
-}
-async function createExtractJob(jobId, userId, startUrl, options) {
-  const db = getDb();
-  await db.execute({
-    sql: `INSERT INTO site_extract_jobs (id, user_id, status, start_url, options, created_at, updated_at)
-          VALUES (?, ?, 'pending', ?, ?, datetime('now'), datetime('now'))`,
-    args: [jobId, userId, startUrl, JSON.stringify(options)]
-  });
-}
-async function getExtractJob(jobId) {
-  const db = getDb();
-  const res = await db.execute({ sql: `SELECT * FROM site_extract_jobs WHERE id = ?`, args: [jobId] });
-  return res.rows[0] ? rowToJob(res.rows[0]) : null;
-}
-async function listExtractJobs(userId) {
-  const db = getDb();
-  const res = await db.execute({ sql: `SELECT * FROM site_extract_jobs WHERE user_id = ? ORDER BY created_at DESC LIMIT 50`, args: [userId] });
-  return res.rows.map((r) => rowToJob(r));
-}
-async function setExtractJobTotal(jobId, totalUrls) {
-  const db = getDb();
-  await db.execute({
-    sql: `UPDATE site_extract_jobs SET status = 'running', total_urls = ?, updated_at = datetime('now') WHERE id = ?`,
-    args: [totalUrls, jobId]
-  });
-}
-async function saveExtractPages(jobId, pages) {
-  if (pages.length === 0) return;
-  const db = getDb();
-  await db.batch(
-    pages.map((p) => ({
-      sql: `INSERT OR REPLACE INTO site_extract_pages (job_id, url, page) VALUES (?, ?, ?)`,
-      args: [jobId, p.url, JSON.stringify({ ...p, bodyMarkdown: "", schema: [] })]
-    }))
-  );
-  await db.execute({
-    sql: `UPDATE site_extract_jobs
-            SET done_urls = (SELECT COUNT(*) FROM site_extract_pages WHERE job_id = ?), updated_at = datetime('now')
-          WHERE id = ?`,
-    args: [jobId, jobId]
-  });
-}
-async function getExtractedPages(jobId) {
-  const db = getDb();
-  const res = await db.execute({ sql: `SELECT page FROM site_extract_pages WHERE job_id = ?`, args: [jobId] });
-  return res.rows.map((r) => JSON.parse(String(r.page)));
-}
-async function countSuccessfulPages(jobId) {
-  const db = getDb();
-  const res = await db.execute({
-    sql: `SELECT COUNT(*) AS n FROM site_extract_pages
-            WHERE job_id = ? AND json_extract(page, '$.status') = 200 AND json_extract(page, '$.wordCount') > 0`,
-    args: [jobId]
-  });
-  return Number(res.rows[0]?.n ?? 0);
-}
-async function getExtractedUrls(jobId) {
-  const db = getDb();
-  const res = await db.execute({ sql: `SELECT url FROM site_extract_pages WHERE job_id = ?`, args: [jobId] });
-  return new Set(res.rows.map((r) => String(r.url)));
-}
-async function completeExtractJob(jobId, artifacts) {
-  const db = getDb();
-  await db.execute({
-    sql: `UPDATE site_extract_jobs SET status = 'complete', artifacts = ?, updated_at = datetime('now') WHERE id = ?`,
-    args: [JSON.stringify(artifacts ?? []), jobId]
-  });
-}
-async function failExtractJob(jobId, error) {
-  const db = getDb();
-  await db.execute({
-    sql: `UPDATE site_extract_jobs SET status = 'failed', error = ?, updated_at = datetime('now') WHERE id = ?`,
-    args: [sanitizeVendorName(error).slice(0, 2e3), jobId]
-  });
-}
-async function settleExtractJob(jobId, userId, refundMc, netChargeMc, reference) {
-  const db = getDb();
-  const job = await getExtractJob(jobId);
-  if (!job || job.billedMc != null) return;
-  if (refundMc <= 0) {
-    await db.execute({ sql: `UPDATE site_extract_jobs SET billed_mc = ?, updated_at = datetime('now') WHERE id = ? AND billed_mc IS NULL`, args: [Math.max(0, netChargeMc), jobId] });
-    return;
-  }
-  await db.batch([
-    { sql: "UPDATE users SET balance_mc = balance_mc + ? WHERE id = ?", args: [refundMc, userId] },
-    { sql: "INSERT INTO ledger (user_id, amount_mc, operation, description) VALUES (?, ?, ?, ?)", args: [userId, refundMc, LedgerOperation.EXTRACT_SITE_REFUND, reference] },
-    { sql: `UPDATE site_extract_jobs SET billed_mc = ?, updated_at = datetime('now') WHERE id = ? AND billed_mc IS NULL`, args: [Math.max(0, netChargeMc), jobId] }
-  ]);
-}
-
-export {
-  createExtractJob,
-  getExtractJob,
-  listExtractJobs,
-  setExtractJobTotal,
-  saveExtractPages,
-  getExtractedPages,
-  countSuccessfulPages,
-  getExtractedUrls,
-  completeExtractJob,
-  failExtractJob,
-  settleExtractJob
-};
-//# sourceMappingURL=chunk-5PZ6N2QM.js.map

package/dist/chunk-5PZ6N2QM.js.map

@@ -1 +0,0 @@
-{"version":3,"sources":["../src/api/site-extract-repository.ts"],"sourcesContent":["import { getDb } from './db.js'\nimport { sanitizeVendorName } from '../errors.js'\nimport { LedgerOperation } from './rates.js'\nimport type { PageData } from './site-extractor.js'\n\nexport type ExtractJobStatus = 'pending' | 'running' | 'complete' | 'failed'\n\nexport interface ExtractJobRow {\n  id: string\n  userId: number | null\n  status: ExtractJobStatus\n  startUrl: string\n  options: Record<string, unknown>\n  totalUrls: number\n  doneUrls: number\n  artifacts: Array<{ key: string; url: string; bytes: number; contentType: string }> | null\n  error: string | null\n  billedMc: number | null\n  createdAt: string\n  updatedAt: string\n}\n\nfunction rowToJob(r: Record<string, unknown>): ExtractJobRow {\n  return {\n    id: String(r.id),\n    userId: r.user_id != null ? Number(r.user_id) : null,\n    status: (r.status != null ? String(r.status) : 'pending') as ExtractJobStatus,\n    startUrl: String(r.start_url ?? ''),\n    options: r.options ? JSON.parse(String(r.options)) : {},\n    totalUrls: Number(r.total_urls ?? 0),\n    doneUrls: Number(r.done_urls ?? 0),\n    artifacts: r.artifacts ? JSON.parse(String(r.artifacts)) : null,\n    error: r.error != null ? String(r.error) : null,\n    billedMc: r.billed_mc != null ? Number(r.billed_mc) : null,\n    createdAt: String(r.created_at ?? ''),\n    updatedAt: String(r.updated_at ?? ''),\n  }\n}\n\nexport async function createExtractJob(jobId: string, userId: number, startUrl: string, options: Record<string, unknown>): Promise<void> {\n  const db = getDb()\n  await db.execute({\n    sql: `INSERT INTO site_extract_jobs (id, user_id, status, start_url, options, created_at, updated_at)\n          VALUES (?, ?, 'pending', ?, ?, datetime('now'), datetime('now'))`,\n    args: [jobId, userId, startUrl, JSON.stringify(options)],\n  })\n}\n\nexport async function getExtractJob(jobId: string): Promise<ExtractJobRow | null> {\n  const db = getDb()\n  const res = await db.execute({ sql: `SELECT * FROM site_extract_jobs WHERE id = ?`, args: [jobId] })\n  return res.rows[0] ? rowToJob(res.rows[0] as unknown as Record<string, unknown>) : null\n}\n\nexport async function listExtractJobs(userId: number): Promise<ExtractJobRow[]> {\n  const db = getDb()\n  const res = await db.execute({ sql: `SELECT * FROM site_extract_jobs WHERE user_id = ? ORDER BY created_at DESC LIMIT 50`, args: [userId] })\n  return res.rows.map(r => rowToJob(r as unknown as Record<string, unknown>))\n}\n\nexport async function setExtractJobTotal(jobId: string, totalUrls: number): Promise<void> {\n  const db = getDb()\n  await db.execute({\n    sql: `UPDATE site_extract_jobs SET status = 'running', total_urls = ?, updated_at = datetime('now') WHERE id = ?`,\n    args: [totalUrls, jobId],\n  })\n}\n\nexport async function saveExtractPages(jobId: string, pages: PageData[]): Promise<void> {\n  if (pages.length === 0) return\n  const db = getDb()\n  await db.batch(\n    pages.map(p => ({\n      sql: `INSERT OR REPLACE INTO site_extract_pages (job_id, url, page) VALUES (?, ?, ?)`,\n      args: [jobId, p.url, JSON.stringify({ ...p, bodyMarkdown: '', schema: [] })],\n    })),\n  )\n  await db.execute({\n    sql: `UPDATE site_extract_jobs\n            SET done_urls = (SELECT COUNT(*) FROM site_extract_pages WHERE job_id = ?), updated_at = datetime('now')\n          WHERE id = ?`,\n    args: [jobId, jobId],\n  })\n}\n\nexport async function getExtractedPages(jobId: string): Promise<PageData[]> {\n  const db = getDb()\n  const res = await db.execute({ sql: `SELECT page FROM site_extract_pages WHERE job_id = ?`, args: [jobId] })\n  return res.rows.map(r => JSON.parse(String((r as unknown as Record<string, unknown>).page)) as PageData)\n}\n\nexport async function countSuccessfulPages(jobId: string): Promise<number> {\n  const db = getDb()\n  const res = await db.execute({\n    sql: `SELECT COUNT(*) AS n FROM site_extract_pages\n            WHERE job_id = ? AND json_extract(page, '$.status') = 200 AND json_extract(page, '$.wordCount') > 0`,\n    args: [jobId],\n  })\n  return Number((res.rows[0] as unknown as Record<string, unknown>)?.n ?? 0)\n}\n\nexport async function getExtractedUrls(jobId: string): Promise<Set<string>> {\n  const db = getDb()\n  const res = await db.execute({ sql: `SELECT url FROM site_extract_pages WHERE job_id = ?`, args: [jobId] })\n  return new Set(res.rows.map(r => String((r as unknown as Record<string, unknown>).url)))\n}\n\nexport async function completeExtractJob(jobId: string, artifacts: ExtractJobRow['artifacts']): Promise<void> {\n  const db = getDb()\n  await db.execute({\n    sql: `UPDATE site_extract_jobs SET status = 'complete', artifacts = ?, updated_at = datetime('now') WHERE id = ?`,\n    args: [JSON.stringify(artifacts ?? []), jobId],\n  })\n}\n\nexport async function failExtractJob(jobId: string, error: string): Promise<void> {\n  const db = getDb()\n  await db.execute({\n    sql: `UPDATE site_extract_jobs SET status = 'failed', error = ?, updated_at = datetime('now') WHERE id = ?`,\n    args: [sanitizeVendorName(error).slice(0, 2000), jobId],\n  })\n}\n\nexport async function settleExtractJob(jobId: string, userId: number, refundMc: number, netChargeMc: number, reference: string): Promise<void> {\n  const db = getDb()\n  const job = await getExtractJob(jobId)\n  if (!job || job.billedMc != null) return\n  if (refundMc <= 0) {\n    await db.execute({ sql: `UPDATE site_extract_jobs SET billed_mc = ?, updated_at = datetime('now') WHERE id = ? AND billed_mc IS NULL`, args: [Math.max(0, netChargeMc), jobId] })\n    return\n  }\n  await db.batch([\n    { sql: 'UPDATE users SET balance_mc = balance_mc + ? WHERE id = ?', args: [refundMc, userId] },\n    { sql: 'INSERT INTO ledger (user_id, amount_mc, operation, description) VALUES (?, ?, ?, ?)', args: [userId, refundMc, LedgerOperation.EXTRACT_SITE_REFUND, reference] },\n    { sql: `UPDATE site_extract_jobs SET billed_mc = ?, updated_at = datetime('now') WHERE id = ? AND billed_mc IS NULL`, args: [Math.max(0, netChargeMc), jobId] },\n  ])\n}\n"],"mappings":";;;;;;;;;;;AAsBA,SAAS,SAAS,GAA2C;AAC3D,SAAO;AAAA,IACL,IAAI,OAAO,EAAE,EAAE;AAAA,IACf,QAAQ,EAAE,WAAW,OAAO,OAAO,EAAE,OAAO,IAAI;AAAA,IAChD,QAAS,EAAE,UAAU,OAAO,OAAO,EAAE,MAAM,IAAI;AAAA,IAC/C,UAAU,OAAO,EAAE,aAAa,EAAE;AAAA,IAClC,SAAS,EAAE,UAAU,KAAK,MAAM,OAAO,EAAE,OAAO,CAAC,IAAI,CAAC;AAAA,IACtD,WAAW,OAAO,EAAE,cAAc,CAAC;AAAA,IACnC,UAAU,OAAO,EAAE,aAAa,CAAC;AAAA,IACjC,WAAW,EAAE,YAAY,KAAK,MAAM,OAAO,EAAE,SAAS,CAAC,IAAI;AAAA,IAC3D,OAAO,EAAE,SAAS,OAAO,OAAO,EAAE,KAAK,IAAI;AAAA,IAC3C,UAAU,EAAE,aAAa,OAAO,OAAO,EAAE,SAAS,IAAI;AAAA,IACtD,WAAW,OAAO,EAAE,cAAc,EAAE;AAAA,IACpC,WAAW,OAAO,EAAE,cAAc,EAAE;AAAA,EACtC;AACF;AAEA,eAAsB,iBAAiB,OAAe,QAAgB,UAAkB,SAAiD;AACvI,QAAM,KAAK,MAAM;AACjB,QAAM,GAAG,QAAQ;AAAA,IACf,KAAK;AAAA;AAAA,IAEL,MAAM,CAAC,OAAO,QAAQ,UAAU,KAAK,UAAU,OAAO,CAAC;AAAA,EACzD,CAAC;AACH;AAEA,eAAsB,cAAc,OAA8C;AAChF,QAAM,KAAK,MAAM;AACjB,QAAM,MAAM,MAAM,GAAG,QAAQ,EAAE,KAAK,gDAAgD,MAAM,CAAC,KAAK,EAAE,CAAC;AACnG,SAAO,IAAI,KAAK,CAAC,IAAI,SAAS,IAAI,KAAK,CAAC,CAAuC,IAAI;AACrF;AAEA,eAAsB,gBAAgB,QAA0C;AAC9E,QAAM,KAAK,MAAM;AACjB,QAAM,MAAM,MAAM,GAAG,QAAQ,EAAE,KAAK,uFAAuF,MAAM,CAAC,MAAM,EAAE,CAAC;AAC3I,SAAO,IAAI,KAAK,IAAI,OAAK,SAAS,CAAuC,CAAC;AAC5E;AAEA,eAAsB,mBAAmB,OAAe,WAAkC;AACxF,QAAM,KAAK,MAAM;AACjB,QAAM,GAAG,QAAQ;AAAA,IACf,KAAK;AAAA,IACL,MAAM,CAAC,WAAW,KAAK;AAAA,EACzB,CAAC;AACH;AAEA,eAAsB,iBAAiB,OAAe,OAAkC;AACtF,MAAI,MAAM,WAAW,EAAG;AACxB,QAAM,KAAK,MAAM;AACjB,QAAM,GAAG;AAAA,IACP,MAAM,IAAI,QAAM;AAAA,MACd,KAAK;AAAA,MACL,MAAM,CAAC,OAAO,EAAE,KAAK,KAAK,UAAU,EAAE,GAAG,GAAG,cAAc,IAAI,QAAQ,CAAC,EAAE,CAAC,CAAC;AAAA,IAC7E,EAAE;AAAA,EACJ;AACA,QAAM,GAAG,QAAQ;AAAA,IACf,KAAK;AAAA;AAAA;AAAA,IAGL,MAAM,CAAC,OAAO,KAAK;AAAA,EACrB,CAAC;AACH;AAEA,eAAsB,kBAAkB,OAAoC;AAC1E,QAAM,KAAK,MAAM;AACjB,QAAM,MAAM,MAAM,GAAG,QAAQ,EAAE,KAAK,wDAAwD,MAAM,CAAC,KAAK,EAAE,CAAC;AAC3G,SAAO,IAAI,KAAK,IAAI,OAAK,KAAK,MAAM,OAAQ,EAAyC,IAAI,CAAC,CAAa;AACzG;AAEA,eAAsB,qBAAqB,OAAgC;AACzE,QAAM,KAAK,MAAM;AACjB,QAAM,MAAM,MAAM,GAAG,QAAQ;AAAA,IAC3B,KAAK;AAAA;AAAA,IAEL,MAAM,CAAC,KAAK;AAAA,EACd,CAAC;AACD,SAAO,OAAQ,IAAI,KAAK,CAAC,GAA0C,KAAK,CAAC;AAC3E;AAEA,eAAsB,iBAAiB,OAAqC;AAC1E,QAAM,KAAK,MAAM;AACjB,QAAM,MAAM,MAAM,GAAG,QAAQ,EAAE,KAAK,uDAAuD,MAAM,CAAC,KAAK,EAAE,CAAC;AAC1G,SAAO,IAAI,IAAI,IAAI,KAAK,IAAI,OAAK,OAAQ,EAAyC,GAAG,CAAC,CAAC;AACzF;AAEA,eAAsB,mBAAmB,OAAe,WAAsD;AAC5G,QAAM,KAAK,MAAM;AACjB,QAAM,GAAG,QAAQ;AAAA,IACf,KAAK;AAAA,IACL,MAAM,CAAC,KAAK,UAAU,aAAa,CAAC,CAAC,GAAG,KAAK;AAAA,EAC/C,CAAC;AACH;AAEA,eAAsB,eAAe,OAAe,OAA8B;AAChF,QAAM,KAAK,MAAM;AACjB,QAAM,GAAG,QAAQ;AAAA,IACf,KAAK;AAAA,IACL,MAAM,CAAC,mBAAmB,KAAK,EAAE,MAAM,GAAG,GAAI,GAAG,KAAK;AAAA,EACxD,CAAC;AACH;AAEA,eAAsB,iBAAiB,OAAe,QAAgB,UAAkB,aAAqB,WAAkC;AAC7I,QAAM,KAAK,MAAM;AACjB,QAAM,MAAM,MAAM,cAAc,KAAK;AACrC,MAAI,CAAC,OAAO,IAAI,YAAY,KAAM;AAClC,MAAI,YAAY,GAAG;AACjB,UAAM,GAAG,QAAQ,EAAE,KAAK,+GAA+G,MAAM,CAAC,KAAK,IAAI,GAAG,WAAW,GAAG,KAAK,EAAE,CAAC;AAChL;AAAA,EACF;AACA,QAAM,GAAG,MAAM;AAAA,IACb,EAAE,KAAK,6DAA6D,MAAM,CAAC,UAAU,MAAM,EAAE;AAAA,IAC7F,EAAE,KAAK,uFAAuF,MAAM,CAAC,QAAQ,UAAU,gBAAgB,qBAAqB,SAAS,EAAE;AAAA,IACvK,EAAE,KAAK,+GAA+G,MAAM,CAAC,KAAK,IAAI,GAAG,WAAW,GAAG,KAAK,EAAE;AAAA,EAChK,CAAC;AACH;","names":[]}