mcp-scraper 0.38.2 → 0.40.1

package/docs/specs/unified-credit-and-scheduled-execution-billing-spec.md

@@ -1,995 +0,0 @@
-# Unified Credits, Integrations, and Scheduled Execution Billing
-
-- Status: Released scheduled-execution contract plus approved connected-account v2 contract
-- Spec date: 2026-07-15
-- Primary repos: mcp-scraper, mcp-scraper-scheduler, mcpscraper-web, mcpscraper-sdk
-- Commercial decision: Scheduled Actions is a product capability, not a separate subscription
-- Connected-account decision: each active Nango connection is a $3/month add-on; usage is metered in Credits
-- Accounting decision: MCP Scraper's main credit ledger is the only debit authority
-
-## Executive decision
-
-MCP Scraper will use one visible usage currency: Credits.
-
-Scheduled Actions uses the same Credit wallet as the rest of MCP Scraper. It is no longer sold as a separate $10-per-month product with a 1,000-run quota. Connected accounts are commercial add-ons: each active Nango connection costs $3 per month, and provider usage is settled from the shared Credit wallet using authoritative receipts.
-
-The replacement contract is:
-
-1. Every paid MCP Scraper plan includes access to scheduling and the ability to add connected accounts.
-2. OAuth connect and reconnect cost 0 Credits. Each active Nango connection is a separate $3-per-month recurring add-on, prorated by the subscription system.
-3. Each scheduled occurrence that passes the balance check and begins execution costs 75 Credits exactly once.
-4. A deterministic `connection_sync` occurrence has no model charge beyond that 75-Credit base.
-5. An agent-mode occurrence additionally settles OpenRouter's actual reported model cost, multiplied by 1.5, then converted to Credits.
-6. Token prices may be shown as current per-million-token estimates in Credits, but OpenRouter's reported cost is the settlement truth.
-7. Separately metered scraper, browser, transcription, video-analysis, and similar tools keep their published rate-card prices when a schedule calls them.
-8. Connected usage costs 2 Credits per Nango function run, 2 Credits per Nango Proxy request, and 5 Credits per compute second, prorated from aggregate milliseconds and rounded once up to integer mc.
-9. The older 5/10/15/25-Credit semantic read/write tariff, per-record/byte tariff, and 20,000-Credit active-connection minimum remain retired and are not enabled.
-
-This design separates the predictable credential-carrying cost from variable work. The $3 recurring line item pays for keeping a Nango connection active; function, Proxy, and compute activity draw from the shared Credit balance. There is no artificial 1,000-run cliff or separate scheduler product.
-
-### Released v1 implementation boundary
-
-- Authorization currently performs an idempotent wallet balance check, not a hard balance reservation.
-- The server-side default maximum is 5,000 Credits per run and is environment-configurable; it is not yet editable in the customer UI.
-- A missing authoritative model cost moves the schedule to `cost_pending` and pauses it. Automated recovery/reconciliation is still a hardening item.
-- The main API accepts idempotent connected-usage receipts and is the settlement authority. The scheduler must preflight at least the fixed 2-Credit function unit, and a prior `cost_pending` connected receipt blocks new work until recovered.
-- User-editable monthly caps, auto-refill, hard reservations, and automated Nango-to-ledger reconciliation remain hardening work.
-
-## What was found before the v1 cutover
-
-The originally intended scheduler economics are recognizable in the code, but the implementation is not safe enough to keep.
-
-### Confirmed historical behavior
-
-- Stripe has a $10-per-month Scheduled Actions checkout.
-- The webhook grants a scheduling entitlement with a 1,000-run period quota.
-- The scheduler contains a 1.5 OpenRouter markup constant.
-- The scheduler reads OpenRouter's provider-reported usage cost.
-- Scheduled agent execution attempts to record model usage and a pass-through cost.
-
-So the answer to “was $10 meant to open the feature and then AI cost was marked up 50%?” is: substantially yes. The entitlement and the 50% markup both exist.
-
-### Pre-cutover billing defect
-
-The pre-cutover path did not reliably implement that intended price:
-
-1. schedule-execute invokes both meterLlmUsage and billPassThroughCost for the generated result.
-2. Both paths can call recordCost.
-3. recordCost can forward each raw AI cost to the main MCP Scraper AI-debit endpoint.
-4. That endpoint applies the existing 3.0 memory-AI multiplier.
-5. If both HTTP debits succeed, the final-step model cost can therefore produce two 3.0 debits—up to 6.0 times that raw cost—while the scheduler's local usage record describes a 1.5 billable amount.
-6. The current metering client is fail-open, so missing configuration or a failed internal request can instead produce no debit.
-7. The current call uses top-level usage and provider metadata. For a multi-step agent result, this describes the last step, not necessarily the complete run. Earlier model steps can be omitted.
-
-The scheduler environment had the internal MCP Scraper URL, internal debit secret, and OpenRouter key configured, so this was a real production risk rather than dead local code. The v1 cutover removed the duplicate/fail-open path and now aggregates model-step costs before one settlement.
-
-### Required immediate interpretation
-
-Do not assume existing scheduled-agent charges equal a clean 1.5 markup. Before changing customers or migrating subscriptions, reconcile scheduler usage records against the main credit ledger. Overcharges receive compensating Credits. Historical fail-open undercharges are not collected retroactively.
-
-## Canonical Build Spec
-
-- **Request:** Keep Scheduled Actions on the shared Credit wallet; charge one 75-Credit occurrence base and correctly settle all-step OpenRouter cost at 1.5; add a $3 active-Nango-connection recurring line item plus receipt-based function, Proxy, and millisecond compute usage.
-- **Lane:** Mastra Agentic Development plus MCP/API product billing.
-- **Capability:** Unified Metered Execution.
-- **Audience:** MCP Scraper customers, scheduled-agent users, MCP clients, SDK users, operators, finance, and support.
-- **User-Facing Outcome:** A user pays $3/month for each active Nango account, sees connected usage itemized in Credits, optionally schedules work, and receives idempotent receipts without duplicate debits on retries.
-- **Target Runtime:** Main MCP Scraper API and credit ledger; Vercel-hosted Mastra scheduler; Inngest workers; Nango connections and Proxy/functions; private Blob artifacts; public MCP; npm/SDK clients.
-- **Inputs:** Authenticated identity, connection ID, tool or sync request, scheduled occurrence, model selection, per-run and monthly Credit limits, actual provider results, OpenRouter usage metadata, bytes and records delivered.
-- **Agents:** Existing Scheduled Action Agent for interpretive work. Deterministic connection_sync workers remain non-agentic. Both use the same billing context.
-- **Workflows:** Connect; direct read/write; bulk export; memory import; schedule estimate; occurrence claim; Credit authorization; execution; incremental settlement; final settlement; receipt; retry/reconciliation.
-- **Tools:** Existing generic connected-service tools, provider-native tools, schedule tools, bulk export/import tools, credits_info, a new schedule estimator, and a shared internal billing adapter.
-- **MCP:** Credit behavior is transport-independent. Tool descriptions state charge categories and estimates, results include receipt summaries, and exact provider tools remain protected by tenant ownership, action switches, and grants.
-- **Signals:** Billing authorization created; connected-use preflight; low balance; budget threshold; cost pending; settlement complete; and insufficient Credits.
-- **Background Tasks:** Usage aggregation, connection-minimum settlement, pending-cost reconciliation, low-balance notifications, automatic refill, legacy subscription migration, receipt retention, and rate-snapshot refresh.
-- **Guardrails:** Integer microcredit arithmetic, idempotent charges, one debit authority, no model-visible billing credentials, fail closed before externally costly work, action approval independent of price, no prompt or provider credential in billing events.
-- **Observability:** Itemized receipt per run; raw vendor cost; multiplier; model/provider; tokens; bytes; changed records; connection and schedule IDs; idempotency keys; authorization, settlement, and refund state.
-- **Enforcement:** A call cannot bypass the billing adapter by using Mastra, the generic MCP bridge, direct REST, SDK, Nango, or a background job. Duplicate occurrence or HTTP retry cannot debit twice.
-- **Persistence:** The main MCP Scraper/Turso credit ledger owns balances and debits. Scheduler Postgres owns schedule state, occurrence state, and a mirrored receipt reference, not the authoritative wallet.
-- **Deployment:** Main API/ledger first, then scheduler and connection adapters, then public MCP, frontend, SDK/npm/MCPB, generated docs, Stripe migration, and Vercel production.
-- **Done Contract:** A production scheduled run charges its 75-Credit base once and aggregates every OpenRouter model step once. Every Nango call is preflighted, reports function/Proxy/compute usage under the active rate revision, and settles once in the main ledger. The UI, MCP, API, SDK, and recurring connection quantity publish the same contract.
-- **Receipt Requirements:** Code and deployment SHAs; rate-table revision; Stripe migration result; test output; production Vercel IDs; npm/MCPB versions; sample agent and deterministic-sync billing receipts; reconciliation report; public pricing and tool-description parity.
-
-## Product contract in plain language
-
-### Scheduling is included
-
-Scheduled Actions remains a first-class page and capability. It is not a separate product, subscription, or quota bundle.
-
-Users with an active paid plan can create schedules immediately. A future pay-as-you-go activation may use a minimum $10 Credit purchase, but that path is not part of the released v1 contract. If released, the $10 must be a wallet purchase rather than an expiring scheduler access fee.
-
-Existing scheduled-action subscriptions are migrated; customers do not keep paying a separate scheduler subscription.
-
-### Customers pay for a run, not internal loops
-
-A scheduled occurrence pays:
-
-    75-Credit execution charge
-    + 1.5 × actual OpenRouter cost when agent mode uses a model
-    + any separately metered scraper, browser, transcription,
-      video-analysis, or other published rate-card tools it calls
-
-Direct connected-service reads, enabled actions, supported exports, and bounded Memory snapshots report the Nango resources they actually consume: function runs, Proxy requests, and function compute milliseconds. The tool's semantic label does not change those infrastructure rates.
-
-Internal pagination, retries, leases, and orchestration steps are not separate “runs.” A retry of the same occurrence cannot pay the fixed charge again.
-
-### OpenRouter AI costs are transparent
-
-The customer-facing rule is:
-
-> AI model use consumes Credits equal to 1.5 times OpenRouter's actual reported cost.
-
-MCP Scraper may show a model's current input, output, cached-input, image, and audio prices as estimated Credits per unit. Actual settlement always uses OpenRouter's returned usage.cost across every successful model step. This correctly captures routing, cache discounts, reasoning tokens, media, and provider-specific pricing.
-
-### Action safety is not for sale
-
-Charging more for a write does not authorize it. Mutations still require:
-
-- a tenant-owned active connection;
-- a live upstream tool that intersects source-controlled policy;
-- the connection's action switch;
-- the exact scheduled grant when scheduled;
-- any required confirmation or destructive-action rule.
-
-## Credit unit and accounting rules
-
-### Canonical units
-
-- Internal accounting unit: mc, an integer microcredit.
-- Public accounting unit: Credit.
-- Conversion: 100 mc = 1 Credit.
-- Billing conversion: 2,000,000 mc per $3.00 of vendor cost, represented as an exact rational rather than a floating-point value.
-- Fractional public Credits may be displayed to two decimal places, but all ledger math uses integer mc.
-- Credit lots continue to expire according to the current three-month policy unless a later commercial decision changes it.
-- Debits consume eligible lots using the existing ledger order.
-
-The public skill currently says 1 Credit equals 1,000 mc while the source rate table says 1 Credit equals 100 mc. This 10-times documentation discrepancy is a release blocker. All public rates and examples must be generated from the canonical source table.
-
-### Current plan grants
-
-The current source rate table implies:
-
-| Plan | Monthly price | Credits | Approximate Credits per dollar |
-|---|---:|---:|---:|
-| Starter | $12 | 80,000 | 6,666.7 |
-| Growth | $40 | 266,667 | 6,666.7 |
-| Scale | $100 | 800,000 | 8,000 |
-
-The Scale grant is a deliberate volume discount. A fixed Credit charge therefore has a lower cash-equivalent markup for a Scale customer. Product copy may say the billing formula is 1.5 times raw OpenRouter cost; it must not promise that every plan produces a 50% cash gross margin after volume discounts.
-
-The billing conversion must become an explicit rate-registry constant. It must no longer be derived from a Stripe plan's price or Credit grant. Otherwise an innocent plan-price edit would silently reprice AI work.
-
-### One-time activation and refill
-
-The old one-time pack mapping must not be reactivated because its $10 pack would grant 110,000 Credits and undercut recurring plans.
-
-Recommended pay-as-you-go contract:
-
-| Purchase | Credits | Purpose |
-|---|---:|---|
-| $10 minimum activation/refill | 50,000 | Enables metered integrations and schedules for a non-plan account |
-| Customer-selected auto-refill | Same 5,000 Credits per dollar | Prevents interrupted schedules, subject to a monthly dollar cap |
-
-Auto-refill is off by default. The customer chooses a threshold and a monthly maximum, and receives a receipt for each Stripe charge.
-
-This is intentionally less generous than subscription grants. Plans remain the best value; pay-as-you-go exists for activation and overflow.
-
-## Released v1 rate table
-
-All rates live in one versioned server-side registry. Frontends, MCP descriptions, SDKs, and docs consume generated projections from that registry.
-
-### Connection operations
-
-Released v1 rates:
-
-| Billing class | Charge |
-|---|---:|
-| OAuth connect, reconnect, or consent callback | Free |
-| Active Nango connection | $3 per month, recurring and prorated |
-| Nango function run | 2 Credits per run |
-| Nango Proxy request | 2 Credits per request |
-| Nango function compute | 5 Credits per second, prorated from aggregate milliseconds |
-
-Action safety remains independent of price: ownership, live inventory, source-controlled policy, the account action switch, and exact scheduled grants still apply.
-
-The active rate revision is `2026-07-15.connected-usage-v1`. Compute settlement uses:
-
-    compute mc = ceil(computeMilliseconds * 500 mc / 1000)
-
-The scheduler aggregates raw milliseconds for one receipt before applying that ceiling. It does not round each span or function up to a whole second. Function, Proxy, and compute line items are additive; the recurring $3 connection charge is not converted into Credits.
-
-### Deferred connected-operation pricing options (not enabled)
-
-Earlier design work proposed 5/10/15/25-Credit read/write classes, a 25-Credit bulk base, per-record charges, and per-MiB charges. Those values are retained only as research for a later commercial decision. They are not present in the production registry and must not appear as current pricing in UI, MCP descriptions, SDKs, or support answers.
-
-### Retired active connection minimum proposal
-
-An earlier proposal described a 20,000-Credit wallet minimum per active connection per 30-day billing period. It is superseded by the flat $3/month recurring add-on and must not be presented as current pricing.
-
-The remainder of this subsection records the deferred proposal for future evaluation only:
-
-One tenant-owned credential/account row is one connection. Two Google accounts, two Slack workspaces, or two WordPress sites are separate connections. Disconnected and billing-paused credentials do not accrue a new minimum; active and reauthorization-grace credentials accrue only for the active seconds in the period.
-
-Eligible connected-service usage offsets this minimum:
-
-- bounded reads and writes;
-- bulk-export base;
-- delivered/changed records;
-- transferred bytes;
-- explicit provider-operation surcharges when later added.
-
-The following do not offset the connection minimum:
-
-- OpenRouter model inference;
-- scheduled-occurrence base;
-- generic web scraping;
-- memory storage capacity;
-- memory embeddings or other AI processing.
-
-The first connection's first 30 days are waived. This creates a real activation path before an idle-connection charge appears.
-
-At settlement, only the shortfall is charged:
-
-    connection minimum debit
-    = max(0, prorated minimum - eligible usage already charged)
-
-If a wallet cannot cover the minimum:
-
-1. Mark the connection billing-paused.
-2. Allow a seven-day read-only recovery window for refill or plan activation.
-3. Block new scheduled use and mutations immediately.
-4. After the grace period, remove or disconnect the stored upstream credential if necessary to stop ongoing vendor charges.
-5. Keep non-secret connection history and explain that reconnection is required.
-
-This behavior must be disclosed before connection and in low-balance notices.
-
-### Scheduled execution
-
-| Billing class | Charge |
-|---|---:|
-| Authorized scheduled occurrence that begins execution | 75 Credits |
-| Duplicate/replayed occurrence | Free |
-| Skipped before execution for insufficient Credits | Free |
-| Work inside the occurrence | Normal applicable rates |
-
-The 75-Credit base preserves the rough economics of the former $10-for-1,000-runs product:
-
-- 1,000 runs use 75,000 Credits.
-- That is approximately $9.38 of Scale plan Credits or $11.25 of Starter/Growth plan Credits.
-
-It also allows a light schedule to remain inexpensive. A schedule running at 8 AM, noon, and 4 PM Mountain executes about 90 times in a 30-day month, using 6,750 Credits in fixed run charges before its data and model work.
-
-#### Agent schedule
-
-An agent occurrence settles:
-
-    75-Credit occurrence base
-    + OpenRouter actual cost times 1.5
-    + any separately metered published tools invoked by the run
-
-#### Deterministic connection sync
-
-A connection_sync occurrence settles:
-
-    75-Credit occurrence base
-
-There is no model charge unless that deterministic pipeline actually invokes a model.
-
-### OpenRouter model inference
-
-The settlement formula is:
-
-    raw_cost_usd = sum(OpenRouter usage.cost for every successful model step)
-    billable_mc = ceil(raw_cost_usd × 1.5 × canonical_mc_per_usd)
-    billable_credits = billable_mc / 100
-
-At the current canonical conversion derived from Starter/Growth grants:
-
-    canonical_mc_per_usd = 2,000,000 / 3
-    $1.00 raw OpenRouter cost = 10,000 billed Credits
-
-As of this spec, OpenRouter lists MiniMax M3 at $0.30 per million input tokens, $1.20 per million output tokens, and $0.06 per million cached-input tokens for standard providers. The customer-facing estimate is therefore:
-
-| MiniMax M3 usage | Estimated Credits |
-|---|---:|
-| 1 million input tokens | 3,000 Credits |
-| 1 million output tokens | 12,000 Credits |
-| 1 million cached-input tokens | 600 Credits |
-
-These are display estimates, not a settlement substitute. The rate UI must show the source timestamp and refresh when OpenRouter pricing changes.
-
-### Worked examples
-
-#### Light scheduled agent
-
-Assume one MiniMax M3 run uses 100,000 input tokens, 20,000 output tokens, and one bounded connected-service read:
-
-| Line item | Credits |
-|---|---:|
-| Scheduled occurrence | 75 |
-| Connected-service read | Included |
-| Estimated model input | 300 |
-| Estimated model output | 240 |
-| Estimated total | 615 |
-
-The model lines settle from the actual reported OpenRouter cost, so 615 is an estimate until the receipt closes.
-
-#### Deterministic bulk sync
-
-Assume a deterministic connection sync delivers 500 changed records and 50 MiB without calling a separately metered tool:
-
-| Line item | Credits |
-|---|---:|
-| Scheduled occurrence | 75 |
-| Connected records and transfer | Included |
-| Total | 75 |
-
-### Existing non-scheduler AI classes
-
-The unified registry keeps existing product-specific multipliers explicit:
-
-| Billing class | Initial multiplier |
-|---|---:|
-| scheduled_agent_llm | 1.5 |
-| memory_ai | 3.0 |
-| site_audit_ai | 1.5 |
-| video_analysis_ai | 1.5 |
-| media_transcription_ai | 3.0 |
-
-The caller supplies the billing class, not the multiplier. The main billing server resolves the multiplier from its versioned registry. This prevents a compromised or outdated worker from choosing its own price.
-
-## Price display policy
-
-The primary display is Credits.
-
-Recommended copy:
-
-> Integrations and scheduling are included. Each run uses Credits for the work it performs.
-
-> AI model use is billed at OpenRouter's actual reported cost plus 50%, converted to Credits.
-
-> Estimated MiniMax M3 rate: 3,000 Credits per million input tokens and 12,000 Credits per million output tokens. Exact usage may vary by provider, caching, reasoning, media, and routing.
-
-The UI may place raw dollar rates in a secondary tooltip or expandable explanation. A single fixed dollar-per-million claim must not replace actual-cost settlement.
-
-## Per-run and monthly budgets
-
-Released v1 has a server-side `maxCreditsPerRun` default of 5,000 Credits and pauses on insufficient balance or unresolved model cost. The cap is environment-configurable and is not yet user-editable. User-defined monthly caps, model allowlists, auto-refill behavior, and a pre-activation estimator remain future work.
-
-Before each occurrence:
-
-1. Perform an idempotent balance authorization before model execution.
-2. If the wallet cannot cover the maximum, skip without a run-base charge.
-3. Settle the 75-Credit base once plus all authoritative model-step cost.
-4. If authoritative model cost is missing, move to `cost_pending` and pause later occurrences.
-
-Hard reservations, incremental budget stopping, monthly caps, and automatic release of unused reserved balance remain hardening work.
-
-The current very large max-output-token ceiling cannot function as the spending guard. Budget enforcement must occur during execution, including after each model step.
-
-## End-to-end execution contract
-
-### Occurrence flow
-
-    Inngest or scheduler trigger
-      -> idempotently claim identity + schedule + scheduledFor
-      -> ask main billing service for an idempotent balance authorization
-      -> receive an opaque billing context
-      -> execute exact schedule grants
-      -> aggregate every OpenRouter model-step cost
-      -> settle the 75-Credit base plus model amount once
-      -> persist billing event references
-
-The billing context travels in an internal authenticated transport header or server-side execution context. It is never put in the model prompt, tool arguments, provider payload, or public result.
-
-Hard reservations, incremental child-call line items, and full customer-facing itemized receipts are the next hardening phase rather than released-v1 claims.
-
-### Direct non-scheduled flow
-
-Direct connected-service MCP, REST, or SDK work is included and still follows the same tenant ownership and permission checks. If that work invokes a separately metered public tool, the existing tool-specific billing path applies:
-
-    validate tenant-owned exact tool
-      -> execute the connected-service operation without an integration debit
-      -> separately meter only published scraper/media/browser classes used
-
-The generic MCP bridge and a provider-native MCP tool must not silently produce different integration pricing for the same operation.
-
-### Multi-step model accounting
-
-For Mastra/AI SDK generate results:
-
-- top-level usage represents the last step;
-- totalUsage aggregates tokens but does not by itself provide a trustworthy total dollar cost;
-- providerMetadata on each step contains that step's OpenRouter usage.cost in the current runtime.
-
-Therefore the scheduler must iterate all result.steps and sum each providerMetadata.openrouter.usage.cost. It must not debit from both a generic usage meter and a pass-through-cost function.
-
-When a step has no usage.cost:
-
-1. Save the OpenRouter generation ID when available.
-2. Query OpenRouter's generation endpoint for the authoritative cost.
-3. Keep the receipt in cost_pending state until reconciled.
-4. Use a current rate snapshot only for authorization and estimates, never silent final settlement.
-5. Pause later occurrences if pending costs cannot be reconciled within the configured window.
-
-## Single debit authority
-
-The main MCP Scraper billing service is the only component allowed to mutate the customer's Credit balance.
-
-| Component | May do | Must not do |
-|---|---|---|
-| Main billing service | Authorize, reserve, settle, refund, expire, and report Credits | Execute schedules or hold provider credentials |
-| Scheduler | Request authorization, emit line items, mirror receipt status | Directly debit a wallet or apply a multiplier |
-| Mastra agent | Use granted tools within a budget | Receive wallet secrets or price itself |
-| Connected-service adapter | Report classified outcomes, records, and bytes | Create plan-dependent rates |
-| Nango | Authenticate provider calls and return results | Own MCP Scraper pricing or Credits |
-| Frontend/SDK/MCP | Display estimates and receipts from server | Calculate authoritative charges locally |
-
-The existing internal memory AI-debit endpoint becomes a compatibility adapter during migration, then routes through the unified billing service. It may not apply an extra 3.0 multiplier to a scheduled_agent_llm event.
-
-## Billing data model and hardening roadmap
-
-The released v1 authorization and billing-event tables are smaller than the target below. Names are illustrative; the following reservation, receipt, and connection-period structures describe future hardening and must not be treated as already released.
-
-### billing_events
-
-Required fields:
-
-- id;
-- identity_id;
-- idempotency_key, globally unique;
-- billing_class;
-- source_surface;
-- schedule_id, nullable;
-- occurrence_id, nullable;
-- connection_id, nullable;
-- tool_name, nullable;
-- model and provider, nullable;
-- generation_id, nullable;
-- raw_cost_usd_nanos or an exact decimal representation, nullable;
-- multiplier_basis_points, nullable;
-- input_tokens, output_tokens, cached_input_tokens, reasoning_tokens, nullable;
-- record_count, nullable;
-- byte_count, nullable;
-- amount_mc;
-- rate_revision;
-- status: estimated, authorized, pending, settled, refunded, void, or cost_pending;
-- safe metadata JSON;
-- created_at and settled_at.
-
-Do not store prompts, model responses, provider tokens, OAuth tokens, email bodies, file content, or raw tool arguments.
-
-### credit_reservations
-
-Required fields:
-
-- id;
-- identity_id;
-- idempotency_key, unique;
-- max_amount_mc;
-- consumed_amount_mc;
-- released_amount_mc;
-- status: active, settled, released, expired;
-- expires_at;
-- occurrence_id or request_id;
-- created_at and updated_at.
-
-The balance available for new work excludes active reservations.
-
-### billing_receipts
-
-A receipt groups line items for one direct request, bulk job slice, or scheduled occurrence:
-
-- receipt ID;
-- identity;
-- source and run IDs;
-- rate revision;
-- total mc and public Credits;
-- line-item summaries;
-- reserved, settled, and released amounts;
-- status;
-- timestamps;
-- safe failure/refund reason.
-
-### connection_billing_periods
-
-Required fields:
-
-- identity and connection ID;
-- period start/end;
-- active seconds and prorated minimum;
-- eligible usage mc;
-- shortfall mc;
-- settlement event ID;
-- status;
-- grace deadline.
-
-## Idempotency and retry rules
-
-### Scheduled occurrence key
-
-The canonical occurrence key is:
-
-    schedule:{identityId}:{scheduleId}:{scheduledForIso}
-
-It owns at most one occurrence-base event. In released v1 the base event is created only after the claim obtains its balance authorization and execution begins. Future hard-reservation work will retain the same key. Worker retries, leases, and resumed slices reference that event.
-
-### Line-item keys
-
-Examples:
-
-    occurrenceKey:base
-    occurrenceKey:llm:{stepId}
-    occurrenceKey:connection:{callId}
-    occurrenceKey:records:{dataset}:{checkpoint}
-    occurrenceKey:bytes:{artifactId}
-    request:{requestId}:tool:{toolCallId}
-
-The main ledger rejects a second settlement for the same key and returns the existing receipt.
-
-### Partial success
-
-- Work completed before a provider or user error remains chargeable.
-- No provider-work line is charged for validation failure before the provider call.
-- Included connected-service records and bytes do not create their own charge in released v1.
-- Unused reservation is always released.
-- The 75-Credit platform base is refunded when MCP Scraper itself fails before useful execution.
-- The base is not refunded for invalid customer configuration, expired OAuth after a valid start, provider outage after dispatch, or user-authored agent failure unless support policy grants a courtesy refund.
-- Actual OpenRouter cost remains chargeable whenever the provider incurred it, even if the final workflow outcome fails.
-
-Refund reasons are structured and support-visible.
-
-## Internal billing API
-
-The exact route naming may follow current conventions, but the contract must include:
-
-### Authorize
-
-POST /api/internal/billing/authorizations
-
-Input:
-
-- identity;
-- idempotency key;
-- billing intent;
-- maximum mc;
-- schedule, occurrence, connection, and request references;
-- rate revision requested or latest.
-
-Output:
-
-- authorization ID;
-- opaque billing-context token;
-- reserved mc;
-- rate revision;
-- expiry;
-- remaining wallet and monthly-budget summaries.
-
-### Add line items
-
-POST /api/internal/billing/authorizations/:id/line-items
-
-Input is a batch of classified, idempotent measurements. The server applies rates and multipliers. The worker never sends an authoritative final Credit amount for a class whose price is server-owned.
-
-### Settle
-
-POST /api/internal/billing/authorizations/:id/settle
-
-The server atomically debits actual mc, releases the remainder, and returns the final receipt.
-
-### Void or refund
-
-POST /api/internal/billing/authorizations/:id/void
-
-POST /api/internal/billing/receipts/:id/refunds
-
-Refunds append compensating ledger events. Settled rows are never silently edited.
-
-### Availability policy
-
-- If authorization is unavailable, fail closed before externally costly work.
-- If line-item reporting is temporarily unavailable after work starts, persist a durable outbox record.
-- If settlement is pending, mark cost_pending and pause future work that could increase exposure.
-- Never discard a charge measurement because an HTTP request failed.
-
-## Public API, MCP, and SDK contract
-
-Released v1 publishes the 75-Credit occurrence base and 1.5-times actual OpenRouter rule across the hosted MCP, npm packages, generated SDK contracts, marketing pages, and scheduler status. The richer fields below are the follow-on receipt and budget roadmap.
-
-### Existing tools to extend
-
-credits_info adds:
-
-- canonical Credit conversion;
-- rate revision;
-- connection rates;
-- connection minimum policy;
-- schedule base;
-- current model estimate table;
-- actual-cost language;
-- credit expiry and auto-refill summary.
-
-create_scheduled_action adds:
-
-- maxCreditsPerRun;
-- maxCreditsPerMonth;
-- estimatedCreditsPerRun;
-- estimate assumptions and warning range.
-
-get_schedule_status adds:
-
-- month-to-date Credit spend;
-- remaining monthly budget;
-- last authorization and receipt IDs;
-- last itemized Credit summary;
-- paused-for-billing state;
-- pending-cost state.
-
-Connection list/status adds:
-
-- current billing period;
-- eligible usage toward minimum;
-- projected shortfall;
-- billing-paused/grace state;
-- next settlement date.
-
-All connected-service execution tools return a compact billing summary:
-
-- receipt ID;
-- Credits charged;
-- line-item category;
-- pending or settled status.
-
-### New estimator
-
-Add estimate_scheduled_action_cost to the root MCP and corresponding REST/SDK methods.
-
-Inputs:
-
-- schedule mode;
-- frequency and timezone;
-- selected connections and tools;
-- expected records/bytes when known;
-- model and estimated input/output tokens when agentic;
-- memory destination;
-- proposed budgets.
-
-Outputs:
-
-- fixed Credits per occurrence and per month;
-- variable estimate range;
-- model rate snapshot;
-- assumptions;
-- recommended maxCreditsPerRun and maxCreditsPerMonth;
-- warning that actual OpenRouter-reported cost controls settlement.
-
-Estimation is free and creates no reservation.
-
-### Tool-description language
-
-Every billable tool schema includes:
-
-- its billing class;
-- fixed charge if any;
-- variable units;
-- whether failure can incur provider/model cost;
-- link or pointer to credits_info;
-- whether it participates in a connection minimum.
-
-Tool descriptions do not hardcode copied rates by hand. They are generated or injected from the canonical registry.
-
-## Mastra integration
-
-### Scheduled Action Agent
-
-The agent receives:
-
-- exact tenant-owned connection toolsets;
-- a server-side Credit budget;
-- a per-step guard;
-- remaining-budget context as trusted runtime metadata, not user prompt content;
-- no access to billing credentials or wallet mutation tools.
-
-Use onStepFinish or the current equivalent to:
-
-1. capture each model step's OpenRouter usage cost;
-2. append one idempotent line item;
-3. check remaining reserved budget;
-4. stop before the next expensive step when the cap is reached.
-
-The final result uses total tool outcomes plus all step records. Remove the dual meterLlmUsage and billPassThroughCost debit behavior.
-
-### Connection sync worker
-
-The deterministic worker:
-
-- claims the same occurrence key;
-- authorizes its budget;
-- charges the 75-Credit occurrence base once;
-- settles observed Nango function runs, Proxy requests, and aggregate compute milliseconds under the active connected-usage rate revision;
-- checkpoints before settlement so retry behavior is deterministic.
-
-The worker never invokes a model merely to paginate or transfer content.
-
-## User experience
-
-### Pricing page
-
-Remove the Scheduled Actions add-on card.
-
-Under every paid plan:
-
-> Integration access and scheduled work included. Each active Nango account is $3/month; connected usage draws from your shared Credit balance.
-
-Show the 75-Credit scheduled occurrence, the “actual OpenRouter cost plus 50%” disclosure, the $3 active-account recurring fee, and the 2/2/5 connected-usage rates. Do not show the retired semantic read/write tariff or active-connection Credit minimum as current pricing.
-
-### Integrations page
-
-Each card shows:
-
-- connected account count;
-- real callable read/action inventory;
-- action switch;
-- a link to import/schedule.
-
-OAuth authorization itself remains free.
-
-### Scheduling page
-
-Keep Scheduling as a prominent feature and navigation destination.
-
-The creation flow shows:
-
-1. frequency and timezone;
-2. selected connection/data/action;
-3. destination or intended result;
-4. estimated fixed and variable Credits;
-5. per-run and monthly caps;
-6. low-balance behavior;
-7. action confirmation/grants;
-8. activate.
-
-The schedule detail view presents an itemized history, not just “one run.”
-
-### Usage page
-
-Group receipts by:
-
-- direct tools;
-- connected-service operations;
-- bulk/import;
-- scheduled occurrences;
-- OpenRouter AI;
-- memory processing;
-- connection minimums;
-- refunds.
-
-Users can filter by connection, schedule, model, provider, and date.
-
-## Scheduled Actions subscription migration
-
-### Live migration result
-
-The legacy live-mode Stripe price `price_1ToXcHS8aAcsk3TGf3cW7zHx` still exists as an active $10 monthly price, but a read-only all-status audit on 2026-07-14 found zero subscriptions. The old checkout route now returns `410 Gone`, so no customer cancellation, proration, or compensating migration Credit is currently required. The price may be archived separately as Stripe catalog cleanup; archiving it is not required for runtime cutover and was not performed automatically.
-
-### Customer treatment
-
-1. Stop new scheduler-subscription checkout after the unified billing path is proven.
-2. Identify active, trialing, past-due, and cancelled-at-period-end scheduler subscriptions.
-3. Grant a migration Credit amount equal to the unused prepaid portion, with a customer-friendly floor.
-4. Cancel the separate Stripe scheduler subscription at the correct boundary without interrupting saved schedules.
-5. Remove entitlement and 1,000-run quota gates.
-6. Preserve schedule definitions, timezone, encrypted service key, grants, and history.
-7. Assign default per-run and monthly budgets.
-8. Notify customers before the first Credit-billed occurrence.
-
-Recommended migration grant:
-
-    max(prorated unused $10 converted at plan rate, 75,000 Credits for a fully unused current period)
-
-The exact grant must be tested against Stripe period data and approved before execution.
-
-### Historical billing reconciliation
-
-Build a one-time report joining:
-
-- scheduler mem_usage_ledger scheduled entries;
-- schedule occurrence/run IDs;
-- raw OpenRouter costs and step metadata where retained;
-- main credit-ledger debit events;
-- Stripe scheduler subscription state.
-
-Classify each occurrence:
-
-- correctly debited once;
-- potentially duplicated;
-- wrong multiplier;
-- missing earlier model steps;
-- fail-open/no debit;
-- no usable evidence.
-
-Issue compensating Credits for defensible overcharges. Do not retroactively debit undercharges. Retain an operator receipt and customer-support note.
-
-## Rollout sequence
-
-### Gate 0 — freeze unsafe expansion
-
-- Do not add more callers to the current scheduled AI debit functions.
-- Add telemetry that identifies both debit paths without changing customer balances.
-- Preserve current schedule service while the replacement is built.
-
-### Gate 1 — canonical registry and generated public rates
-
-- Move Credit conversion, billing classes, multipliers, and display estimates into one versioned registry.
-- Generate the MCP skill, public rate API, frontend copy data, and SDK constants.
-- Fix the 100-versus-1,000 mc public discrepancy.
-
-### Gate 2 — unified ledger and reservation API
-
-- Add billing events, reservations, receipts, idempotency constraints, and durable outbox.
-- Wrap existing direct Credit debits behind the service.
-- Keep old endpoints as compatibility adapters.
-
-### Gate 3 — shadow scheduler billing
-
-- Authorize no real reservations yet.
-- Calculate the proposed 75-Credit base and all-step OpenRouter amount in shadow.
-- Compare against current debits and production receipts.
-- Prove multi-step aggregation and single-debit behavior.
-
-### Gate 4 — scheduler cutover
-
-- Enable reservations and settlement for internal/test tenants.
-- Remove the duplicate model debit path.
-- Enable per-run/month caps and cost-pending reconciliation.
-- Expand to production cohorts after receipt review.
-
-### Gate 5 — deterministic sync and direct connection billing (partially deferred)
-
-- Route `connection_sync` through the shared 75-Credit occurrence context.
-- Keep generic MCP, REST, SDK, exports, and snapshots included in released v1.
-- Reconsider connected-operation tariffs or active-connection minimums only after a new explicit commercial decision, shadow period, and customer notice.
-
-### Gate 6 — commercial migration
-
-- Stop new schedule add-on sales.
-- Grant migration Credits and cancel old Stripe subscriptions.
-- Remove entitlement/quota enforcement.
-- Activate $10 pay-as-you-go refill only after its Stripe and ledger path is complete.
-
-### Gate 7 — full release
-
-- Update frontend, pricing, scheduler copy, MCP schemas, npm package, SDKs, MCPB, terms, support docs, and Vercel production.
-- Run production smoke receipts.
-- Publish a customer explanation and migration FAQ.
-
-## Required tests
-
-### Rate and arithmetic tests
-
-- 100 mc equals 1 public Credit everywhere.
-- All charges use integers and defined ceiling rules.
-- Vendor-cost conversion uses exact rational or decimal arithmetic, never binary floating-point ledger math.
-- OpenRouter raw cost times 1.5 yields the exact expected mc.
-- Plan grants do not alter the number of Credits charged for an identical action.
-- MiniMax display estimates are generated from a timestamped rate snapshot.
-
-### Model accounting tests
-
-- One-step result settles once.
-- Multi-step result sums every step's usage.cost.
-- Top-level final-step metadata alone cannot settle the run.
-- meterLlmUsage and pass-through code cannot each debit the same result.
-- Missing cost becomes cost_pending and reconciles by generation ID.
-- Model failure after incurring cost retains that provider-cost line.
-
-### Scheduling tests
-
-- A new schedule does not require a scheduler subscription.
-- One occurrence receives one 75-Credit base.
-- Retry and lease replay do not duplicate the base.
-- Insufficient Credits before authorization produces a free skip.
-- Per-run cap stops later model/provider work.
-- Monthly cap pauses future occurrences.
-- Unused reservation is released.
-- agent and connection_sync modes both produce receipts.
-- Existing saved schedules survive entitlement migration.
-
-### Connection tests
-
-- OAuth callback is free.
-- Direct reads, enabled actions, exports, and bounded Memory snapshots do not create an integration-operation debit.
-- Internal pagination is not separately billed.
-- Direct MCP, REST, SDK, Mastra, and background calls publish the same included-use rule.
-- Deferred operation tariffs and active-connection minimums are absent from the production registry and public copy.
-
-### Ledger tests
-
-- Globally unique idempotency keys prevent double debit.
-- Reservation and settlement are atomic.
-- Refunds append compensating events.
-- Active reservations reduce available balance.
-- Durable outbox survives billing-service interruption.
-- No prompt, result body, OAuth token, or billing secret enters billing metadata.
-
-### Product-surface tests
-
-- Pricing page no longer sells Scheduled Actions separately.
-- Integrations and Scheduling are in desktop and mobile navigation.
-- credits_info, estimator, schedule status, receipts, UI, and SDK agree.
-- Public skill and source rate table have generated parity.
-- npm, MCPB, hosted MCP, REST, and Vercel production expose the same schemas.
-
-## Production proof
-
-Before marking complete, capture:
-
-1. A tiny direct connected-service read and receipt.
-2. A safe approved connected-service write and receipt.
-3. A two-step scheduled agent run proving both OpenRouter step costs settle once.
-4. A deterministic sync with unchanged and changed records proving delta billing.
-5. An insufficient-balance occurrence proving a free skip.
-6. A retry proving no duplicate base or line item.
-7. A failed MCP Scraper execution proving the correct base refund.
-8. A current model-rate endpoint and matching frontend/MCP display.
-9. A migrated scheduler subscription with saved schedule continuity.
-10. Vercel deployment, npm version, MCPB manifest, SDK version, and generated-doc parity.
-
-Receipts must be sanitized. Do not include prompts, connection tokens, API keys, customer content, raw provider arguments, or signed artifact URLs.
-
-## Release blockers
-
-- Any remaining path that directly mutates Credits outside the main billing service.
-- Any duplicate scheduled model debit.
-- Settlement based only on final-step metadata.
-- Fail-open execution that can incur provider cost without a durable pending charge.
-- Missing idempotency across occurrence, tool call, record batch, or artifact.
-- The public 1 Credit equals 1,000 mc mismatch.
-- Separate scheduling entitlement or 1,000-run quota still enforced on any surface.
-- Frontend, hosted MCP, SDK/npm, MCPB, or public docs showing different rates.
-- A fixed token rate described as authoritative rather than an estimate.
-- Any surface that omits or contradicts the approved $3 plus 2/2/5 connected-account contract.
-- Commercial migration without a reconciliation and compensating-credit policy.
-
-## Decisions locked by this spec
-
-- Scheduled Actions is included and metered, not a separate subscription.
-- Credits are the primary customer-facing unit.
-- OpenRouter actual reported cost is the AI settlement source.
-- scheduled_agent_llm uses a 1.5 multiplier.
-- Model token tables are estimates.
-- The fixed scheduled-occurrence charge is 75 Credits.
-- Active Nango connections cost $3/month; usage costs 2 Credits/function execution, 2 Credits/Proxy request, and 5 Credits/compute-second measured from milliseconds.
-- The proposed 5/10/15/25 semantic tariffs and 20,000-Credit active-connection minimum are not enabled.
-- OAuth is free.
-- Main MCP Scraper owns the only mutable Credit ledger.
-- Every product and execution surface publishes the same price contract; shared itemized receipt UX remains hardening work.
-- Action authorization remains separate from billing.
-
-## Decisions that may change only through a rate revision
-
-- Exact semantic per-read, per-write, per-record, and per-MiB rates beyond the current Nango resource meter.
-- Connected-account monthly price and function, Proxy, or compute rates.
-- Pay-as-you-go Credit grant per dollar.
-- Default per-run and monthly caps.
-- Existing non-scheduler AI multipliers.
-
-Changing one of these does not require a protocol redesign, but it requires a new rate revision, generated-surface parity, tests, and customer notice where applicable.
-
-## Source references
-
-- OpenRouter usage accounting: https://openrouter.ai/docs/cookbook/administration/usage-accounting
-- OpenRouter API usage object: https://openrouter.ai/docs/api/reference/overview
-- Current MiniMax M3 pricing: https://openrouter.ai/minimax/minimax-m3/pricing
-- Current MCP Scraper rate source: src/api/rates.ts
-- Current main AI debit route: src/api/server.ts
-- Current scheduler billing logic: ../mcp-scraper-scheduler/src/mastra/lib/schedule-billing.ts
-- Current scheduled execution path: ../mcp-scraper-scheduler/src/mastra/steps/schedule-execute.ts
-- Current schedule entitlement re-export: ../mcp-scraper-scheduler/src/mastra/db/schedule-entitlements.ts
-- Current scheduler usage re-export: ../mcp-scraper-scheduler/src/mastra/db/usage.ts
-- Installed memory DB entitlement implementation: ../mcp-scraper-scheduler/node_modules/mcpscraper-memory-db/dist/db/schedule-entitlements.js
-- Installed memory DB usage and debit implementation: ../mcp-scraper-scheduler/node_modules/mcpscraper-memory-db/dist/db/usage.js
-
-## Final acceptance statement
-
-The implementation is complete only when a customer can connect a service, schedule or directly invoke work, move data into memory, and receive one understandable Credit receipt whose amount is identical across the live ledger, frontend, MCP, REST, Mastra, and SDK. A local code path, pricing-page change, or scheduler-only implementation does not satisfy this spec.