mcp-scraper 0.38.2 → 0.40.1

package/docs/specs/connected-services-control-plane-decoupling-spec.md

@@ -1,1044 +0,0 @@
-# Connected Services Control Plane Decoupling
-
-- Status: Superseded on 2026-07-16 by `main-mcp-integration-ownership-spec.md`
-- Spec date: 2026-07-16
-- Incident driver: Search Console connection displayed `Connected` while every live read and export returned `connection_transport_unavailable`
-- Primary repos: `mcp-scraper`, `mcp-scraper-scheduler`
-- Adjacent release surfaces: Vercel production projects, root MCP tool contract, generated frontend bundle, billing ledger, Nango Cloud
-- Supersedes: using the scheduler deployment as the unnamed runtime for direct connected-service operations
-- Preserves: the connected-account and usage-pricing decisions in `unified-credit-and-scheduled-execution-billing-spec.md`
-
-> Supersession note: this document correctly identified the hidden scheduler dependency and the need to separate lifecycle from operational health, but it placed integration execution in a separate control-plane deployment. Product ownership requires integrations to live on the main MCP. The replacement specification moves connection storage, credentials, provider execution, health, lifecycle controls, and connected-usage settlement into `mcp-scraper`; the Mastra scheduler remains an independent consumer.
-
-## Executive decision
-
-MCP Scraper will separate **connected-service execution** from **scheduled execution**.
-
-The system needs a secure service that owns tenant connection routing, Nango credentials, live tool discovery, provider calls, connection lifecycle operations, usage metering, and operational health. That service is not conceptually a scheduler and must not share the scheduler's deployment availability boundary.
-
-Create a dedicated **Connected Services Control Plane** deployment. The first implementation is extracted from `mcp-scraper-scheduler` into a reusable integration module and deployed as an independent Vercel project. This avoids moving credentials into the public API and avoids migrating the existing Postgres connection records.
-
-The resulting call paths are:
-
-```text
-Immediate connected read/export/action
-  MCP client or dashboard
-    -> MCP Scraper main API
-    -> Connected Services Control Plane
-    -> Nango
-    -> provider
-
-Scheduled connected work
-  Inngest occurrence
-    -> MCP Scraper Scheduler
-    -> shared integration module / Connected Services Control Plane
-    -> Nango
-    -> provider
-```
-
-The scheduler remains necessary only for recurring jobs, leases, checkpoints, deterministic sync, and scheduled-agent execution. It is not required for one-time Search Console analysis.
-
-## Incident statement
-
-On 2026-07-16, the tenant Search Console connection had all of the following stored metadata:
-
-- provider key `google-search-console`;
-- connection status `connected`;
-- `reconnectRequired: false`;
-- a non-empty read-tool inventory;
-- the expected Search Console connection reference.
-
-Despite that metadata, two `list-sites` attempts and a bounded 90-day `search_console_performance` export failed with retryable HTTP 503 responses:
-
-```json
-{
-  "code": "connection_transport_unavailable",
-  "status": 503,
-  "retryable": true,
-  "message": "The service connection transport is temporarily unavailable."
-}
-```
-
-Public scraper operations such as SERP search and URL extraction remained healthy. The failure was therefore isolated to the connected-service execution path, not the root MCP Scraper transport.
-
-The production Integrations page also displayed the connection as `Connected · Ready for sync`. Its drawer exposed `Refresh access` but no `Disconnect` control. The deployed frontend bundle rendered Disconnect only for `remote_mcp` connections. Current local source removes that transport restriction, proving a production bundle/source mismatch.
-
-## Root causes
-
-### R1. Stored lifecycle state is presented as runtime health
-
-`Connected` currently means that a tenant-owned connection row exists, is marked active, and is not flagged for reauthorization. It does not mean that a provider call has succeeded recently.
-
-The frontend derives readiness from stored status and approved-tool inventory. It does not distinguish:
-
-- OAuth credential lifecycle;
-- control-plane availability;
-- Nango MCP availability;
-- live provider authorization;
-- provider rate limiting;
-- last successful operation.
-
-### R2. Direct operations depend on the scheduler deployment
-
-`mcp-scraper/src/api/nango-control.ts` defaults `NANGO_CONTROL_URL` to `https://mcp-scraper-scheduler.vercel.app`. The main API forwards connect, reconnect, disconnect, list, describe, read, action, binding, and export-page operations to internal routes in that deployment.
-
-The scheduler deployment therefore acts as both:
-
-1. a recurring-work scheduler; and
-2. the Nango integration gateway for all direct operations.
-
-The second role is real but unnamed, hidden from the product model, and coupled to scheduler runtime health.
-
-### R3. Frontend and backend releases can drift
-
-The local `public/app.jsx` and generated `public/app.js` expose Disconnect for every connection transport. The deployed bundle contains the earlier `remote_mcp`-only condition. The release flow does not currently fail when the public bundle, server routes, and production deployment do not share the intended connection-management contract.
-
-### R4. Provider policy is duplicated
-
-Read/action allowlists and connection-sync requirements exist across the main and scheduler repositories. Comments instruct maintainers to keep the copies aligned, but the contract is not generated from one canonical source. Tool count and readiness can therefore drift across the UI, root MCP, scheduler, and live provider inventory.
-
-## Goals
-
-1. Make one-time connected-service reads independent of scheduler deployment health.
-2. Preserve Nango as the OAuth credential and token-refresh boundary.
-3. Preserve the existing tenant-owned connection references and Postgres records.
-4. Report credential state and operational state separately.
-5. Never label a connection `Ready` solely because a row and allowlist exist.
-6. Provide reliable connect, reconnect, test, and disconnect controls for Nango and remote-MCP connections.
-7. Keep billing, action safety, tenant isolation, audit, and idempotency intact.
-8. Give operators request-level evidence across the main API, control plane, Nango, and provider.
-9. Prevent source/bundle/deployment contract drift.
-10. Cut over without interrupting existing schedules or requiring customers to reconnect healthy accounts.
-
-## Non-goals
-
-- Replacing Nango as the OAuth store.
-- Changing the approved $3-per-active-Nango-account commercial contract.
-- Changing connected function, Proxy, or compute rates.
-- Moving scheduled occurrence, lease, checkpoint, or sync execution into the main API.
-- Exposing provider credentials or Nango connection IDs to browsers or MCP clients.
-- Renaming existing Postgres tables during the first cutover.
-- Building a provider-specific Search Console agent inside the control plane.
-- Treating health checks as proof that every accessible Search Console property has data.
-
-## Terminology
-
-| Term | Meaning |
-|---|---|
-| Connection lifecycle | Whether an OAuth/remote-MCP credential record exists, is active, requires reauthorization, is pending deletion, or is deleted |
-| Operational health | Whether the system has recently reached the connection transport and provider successfully |
-| Control plane | Tenant routing, live tool discovery, provider execution, lifecycle mutation, health evidence, audit, and metering |
-| Scheduler | Recurring occurrence ownership, leases, background execution, checkpoints, and scheduled receipts |
-| Main API | Public authenticated REST/MCP facade, user entitlement enforcement, artifact delivery, and authoritative Credit ledger |
-| Provider contract | Canonical source-controlled classification, required scopes/features, safe probes, action policy, and sync requirements |
-
-## Current architecture
-
-### Main API responsibilities today
-
-The main API currently owns:
-
-- customer/API-key authentication;
-- plan gating;
-- public `/schedule-connections/*` routes;
-- root MCP bridge tools;
-- bounded export orchestration and private artifact delivery;
-- connected-account quantity reconciliation;
-- authoritative Credit debits.
-
-For Nango connections, it does not execute provider tools itself. It calls `controlRequest()` in `src/api/nango-control.ts`, using `NANGO_CONTROL_URL` and `SCHEDULE_INTEGRATIONS_SECRET`.
-
-### Scheduler responsibilities today
-
-The scheduler repository currently owns:
-
-- `mem_external_connections` and connection-related Postgres tables;
-- connection-to-schedule bindings;
-- Nango connect and reconnect sessions;
-- deletion and action switches;
-- provider policy maps;
-- live Nango MCP discovery;
-- direct read and action execution;
-- connected-data export pages;
-- Nango usage metering and action audit;
-- deterministic scheduled sync and agent-mode scheduled execution.
-
-### Credential boundary
-
-Nango owns provider credentials and token refresh. Postgres stores tenant identity, provider key, Nango routing identifier, display metadata, lifecycle state, action switch, table/vault references, and timestamps. It intentionally stores no OAuth token.
-
-This boundary remains correct and must be preserved.
-
-## Target architecture
-
-### Component boundaries
-
-| Component | Owns | Must not own |
-|---|---|---|
-| MCP Scraper main API | Public auth, entitlement checks, stable REST/MCP facade, authoritative Credit ledger, private artifact authorization, aggregate export response | Provider tokens, Nango secret, scheduler leases |
-| Connected Services Control Plane | Connection rows, Nango credential routing, provider contracts, live discovery, reads/actions, export pages, lifecycle operations, operational health evidence, audit, connected-usage receipts | Public customer authentication, model orchestration, schedule occurrence ownership |
-| MCP Scraper Scheduler | Recurrence, leases, execution budgets, deterministic sync, checkpoints, scheduled-agent runs, run receipts | Public connection-management facade, independent provider policy copies |
-| Nango | OAuth consent, encrypted provider credentials, refresh, MCP/Proxy/functions | Product tenancy, product billing, schedule state |
-| Main Credit ledger | Authoritative balance, idempotent debit/credit receipts | Connection credentials, provider payloads |
-| Private artifact storage | Owner-scoped JSONL and export artifacts | Provider credentials, authorization decisions |
-
-### Deployment topology
-
-Deploy the Connected Services Control Plane independently, initially from the scheduler repository:
-
-```text
-source repo: mcp-scraper-scheduler
-shared module: src/mastra/integrations/*
-deployment A: integrations.mcpscraper.dev
-deployment B: mcp-scraper-scheduler.vercel.app
-```
-
-Deployment A exposes only internal integration-control routes and health endpoints. Deployment B exposes scheduled workflow routes and workers. Both import the same provider contracts, connection repository, Nango client, usage metering, and error mapping from the shared module.
-
-This is preferable to copying the code into the main repository because:
-
-- the Nango secret remains outside the public API deployment;
-- the existing Postgres connection repository remains authoritative;
-- scheduler and direct operations consume the same code rather than two policy copies;
-- cutover needs no credential or connection-row migration.
-
-A future repository extraction is allowed only after the shared module and deployment boundary are stable. It is not part of this release.
-
-## Architectural decisions
-
-### D1. Separate lifecycle state from operational health
-
-Every public connection summary must expose both:
-
-```ts
-type ConnectionLifecycleStatus =
-  | 'pending_oauth'
-  | 'connected'
-  | 'reauth_required'
-  | 'deleting'
-  | 'disconnected'
-
-type ConnectionOperationalStatus =
-  | 'unknown'
-  | 'available'
-  | 'temporarily_unavailable'
-  | 'rate_limited'
-  | 'permission_limited'
-  | 'provider_error'
-```
-
-`connected` is never rendered as `Ready` unless recent operational evidence supports it. A connection that has never been used is `Connected · Not tested`.
-
-### D2. Health is evidence from operations, not a page-load side effect
-
-Loading the Integrations page must not call every provider. That would create latency, cost, rate pressure, and noisy failures.
-
-Operational health is updated by:
-
-- successful or failed real reads, actions, descriptions, exports, and scheduled syncs;
-- one provider-specific verification after a successful connect/reconnect callback;
-- an explicit user-triggered Test connection operation.
-
-The UI displays the timestamp and source of the latest evidence. Evidence expires to `unknown` after a provider-contract TTL, default six hours for ordinary connections and one hour after a failure.
-
-### D3. A 503 is not a reconnect signal
-
-Only authentication/authorization evidence can set `reauth_required`, such as a confirmed provider 401, invalid grant, revoked token, or Nango auth-error state.
-
-Transport 5xx, Nango MCP discovery failure, billing-control unavailability, and timeouts set operational state but do not change credential lifecycle. The UI must offer Retry/Test, not Reconnect, for those errors.
-
-### D4. Provider contracts have one canonical source
-
-Create one versioned provider-contract module in the scheduler repository and generate projections for the main repository.
-
-Each provider contract contains:
-
-```ts
-interface ConnectedProviderContract {
-  providerConfigKey: string
-  reads: readonly string[]
-  actions: readonly string[]
-  adminDenylist: readonly string[]
-  requiredPermissionsByTool: Record<string, readonly string[]>
-  requiredFeaturesByTool: Record<string, readonly string[]>
-  sync?: {
-    requiredTools: readonly string[]
-    optionalTools: readonly string[]
-  }
-  healthProbe?: {
-    tool: string
-    args: Record<string, unknown>
-    successPredicate: string
-    ttlSeconds: number
-  }
-}
-```
-
-The generated JSON projection includes a schema version and SHA-256 hash. Main API builds fail when its generated projection differs from the scheduler source.
-
-### D5. Existing public routes remain compatible during migration
-
-Introduce clearer public routes under `/integrations` while retaining `/schedule-connections` aliases for at least one published MCP/package compatibility window.
-
-New clients use:
-
-```text
-GET    /integrations
-POST   /integrations/connect-session
-POST   /integrations/:id/reconnect-session
-POST   /integrations/:id/test
-DELETE /integrations/:id
-POST   /integrations/:id/actions-enabled
-POST   /integrations/:id/read
-POST   /integrations/:id/describe
-POST   /integrations/:id/action
-POST   /integrations/:id/export
-```
-
-The root MCP tool names remain stable in this release. Only their descriptions and result schema gain health information.
-
-### D6. Internal control calls are authenticated and traceable
-
-Replace the generic shared-bearer-only contract with signed service requests:
-
-- service identity;
-- request ID;
-- issued-at timestamp;
-- short expiry;
-- method and path;
-- SHA-256 body digest;
-- HMAC signature using a rotatable internal secret.
-
-Accept the old bearer secret during cutover only. Never forward customer-provided identity headers directly. The main API derives tenant identity from its authenticated user and signs that identity into the internal request body.
-
-### D7. Direct and scheduled paths share execution code
-
-The control plane and scheduler import the same functions for:
-
-- connection ownership resolution;
-- provider-policy intersection;
-- Nango client creation;
-- schema discovery;
-- permission/feature verification;
-- read/action execution;
-- export page traversal;
-- usage measurement and receipts;
-- error normalization;
-- operational-health recording.
-
-Scheduled execution adds schedule grants and occurrence context around the shared executor. It does not maintain a separate provider implementation.
-
-### D8. Disconnect is a first-class lifecycle operation
-
-Disconnect must be visible for every tenant-owned connection transport.
-
-The operation is ordered as follows:
-
-1. authenticate the tenant and resolve ownership;
-2. create an idempotent deletion receipt and mark lifecycle `deleting`;
-3. revoke/delete the credential in Nango or the remote-MCP store;
-4. delete the local connection row in a transaction;
-5. cascade schedule bindings, checkpoints, synced records, and action audit according to retention policy;
-6. return the remaining billable Nango quantity;
-7. reconcile Stripe quantity from the main API;
-8. mark the deletion receipt complete.
-
-If credential deletion is unavailable, retain the local row in `deleting` and return a retryable error. Do not silently delete local routing while leaving an active credential. An administrative force-local-delete path is separate, audited, and unavailable to ordinary users.
-
-### D9. Reconnect updates the same logical connection
-
-Reconnect must preserve the public connection reference, schedule bindings, billing identity, table/vault associations, and audit history. It refreshes or replaces the upstream Nango credential routing identifier only if Nango requires it.
-
-Reconnect must not create an additional billable connection for the same logical account. Duplicate detection uses tenant, provider key, and upstream connection identity inside the control plane.
-
-### D10. Reads retry safely; actions require idempotency
-
-The control plane may retry read-only discovery/read/export-page operations for retryable transport failures with bounded exponential backoff and jitter:
-
-- maximum two retries after the original attempt;
-- total retry budget no greater than the public request deadline;
-- respect provider `Retry-After`;
-- do not retry validation, permission, or authentication failures.
-
-Actions are never automatically replayed after an ambiguous provider outcome. They retain the existing idempotency/outcome-unknown contract.
-
-### D11. Release identity is part of the contract
-
-Every deployment returns:
-
-```json
-{
-  "service": "main-api | integrations-control | scheduler",
-  "releaseId": "git-sha-or-build-id",
-  "providerContractVersion": "...",
-  "providerContractHash": "...",
-  "builtAt": "RFC3339"
-}
-```
-
-The frontend embeds its release ID. Production verification fails if the UI bundle lacks a control promised by the server contract or if provider-contract hashes differ.
-
-### D12. Search Console uses a bounded safe diagnostic
-
-The Search Console provider contract defines:
-
-```json
-{
-  "tool": "list-sites",
-  "args": {},
-  "successPredicate": "result contains a valid site collection, including an empty collection",
-  "ttlSeconds": 21600
-}
-```
-
-A successful probe proves that MCP Scraper reached Nango, Google accepted the credential, and Search Console returned a valid property collection. It does not prove that a named property exists or has performance rows.
-
-After the generic probe, the Geny's Flowers workflow must separately:
-
-1. match an accessible property by normalized domain/site URL;
-2. report the exact selected property;
-3. run a bounded Search Analytics query;
-4. only then begin page extraction, SERP, PAA, or strategy synthesis.
-
-## Public connection representation
-
-The connection summary becomes:
-
-```ts
-interface PublicServiceConnection {
-  connectionId: string
-  providerConfigKey: string
-  provider: string | null
-  label: string
-  transport: 'nango' | 'remote_mcp'
-
-  lifecycleStatus: ConnectionLifecycleStatus
-  reconnectRequired: boolean
-
-  operationalStatus: ConnectionOperationalStatus
-  lastCheckedAt: string | null
-  lastSuccessfulCallAt: string | null
-  lastFailureAt: string | null
-  lastFailureCode: string | null
-  lastFailureRetryable: boolean | null
-  healthEvidenceSource: 'connect_probe' | 'manual_test' | 'direct_read' | 'action' | 'export' | 'scheduled_sync' | null
-
-  actionsEnabled: boolean
-  readTools: string[]
-  actionTools: string[]
-  toolCapabilities: ToolCapability[]
-  tableName: string | null
-  createdAt: string | null
-  updatedAt: string | null
-}
-```
-
-Compatibility fields `status` and `reconnectRequired` remain for one version. `status` maps only to lifecycle and must be documented as such.
-
-## Error contract
-
-All surfaces use the same safe taxonomy:
-
-| Code | HTTP | Lifecycle effect | Operational effect | User action |
-|---|---:|---|---|---|
-| `connection_not_found` | 404 | none | none | Select another connection |
-| `connection_inactive` | 409 | reauth when confirmed | unknown | Reconnect |
-| `provider_auth_revoked` | 401/409 | `reauth_required` | permission limited | Reconnect |
-| `missing_provider_permission` | 403 | connected | permission limited | Reconnect with expanded scope when supported |
-| `connection_transport_unavailable` | 503 | connected | temporarily unavailable | Retry/test later; operator checks control plane |
-| `tool_discovery_failed` | 502 | connected | provider error | Retry; inspect Nango discovery |
-| `upstream_rate_limited` | 429 | connected | rate limited | Retry after returned time |
-| `provider_error` | 502 | connected | provider error | Retry or inspect provider status |
-| `control_plane_not_configured` | 503 | connected | temporarily unavailable | Operator configuration fix |
-| `billing_authorization_unavailable` | 503 | connected | temporarily unavailable | Retry; do not reconnect |
-| `actions_disabled` | 403 | connected | unchanged | Explicitly enable actions |
-| `action_outcome_unknown` | 409 | connected | unchanged | Verify provider state before another action |
-
-Every error includes `requestId`, `retryable`, `sourceLayer`, and a safe message. Provider tokens, raw payloads, customer prompts, and secrets never appear in errors.
-
-## Persistence changes
-
-### Extend connection records
-
-Add to `mem_external_connections`:
-
-```sql
-ALTER TABLE mem_external_connections
-  ADD COLUMN IF NOT EXISTS lifecycle_status TEXT NOT NULL DEFAULT 'connected',
-  ADD COLUMN IF NOT EXISTS operational_status TEXT NOT NULL DEFAULT 'unknown',
-  ADD COLUMN IF NOT EXISTS last_checked_at TIMESTAMPTZ,
-  ADD COLUMN IF NOT EXISTS last_successful_call_at TIMESTAMPTZ,
-  ADD COLUMN IF NOT EXISTS last_failure_at TIMESTAMPTZ,
-  ADD COLUMN IF NOT EXISTS last_failure_code TEXT,
-  ADD COLUMN IF NOT EXISTS last_failure_retryable BOOLEAN,
-  ADD COLUMN IF NOT EXISTS health_evidence_source TEXT,
-  ADD COLUMN IF NOT EXISTS consecutive_failures INTEGER NOT NULL DEFAULT 0;
-```
-
-Keep the existing `status` column during migration. Backfill `lifecycle_status` from it, then make new code write both until every deployed reader supports the new column.
-
-### Deletion receipts
-
-Add an idempotent receipt table:
-
-```sql
-CREATE TABLE IF NOT EXISTS mem_external_connection_deletions (
-  idempotency_key TEXT PRIMARY KEY,
-  identity TEXT NOT NULL,
-  external_connection_id TEXT NOT NULL,
-  provider_config_key TEXT NOT NULL,
-  status TEXT NOT NULL,
-  upstream_deleted_at TIMESTAMPTZ,
-  local_deleted_at TIMESTAMPTZ,
-  remaining_billable_connections INTEGER,
-  error_code TEXT,
-  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
-  updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
-);
-```
-
-The receipt contains no token or provider content.
-
-### Health events
-
-Persist bounded diagnostic history separately from the summary row:
-
-```sql
-CREATE TABLE IF NOT EXISTS mem_external_connection_health_events (
-  id TEXT PRIMARY KEY,
-  identity TEXT NOT NULL,
-  external_connection_id TEXT NOT NULL,
-  operation TEXT NOT NULL,
-  source_surface TEXT NOT NULL,
-  status TEXT NOT NULL,
-  error_code TEXT,
-  retryable BOOLEAN,
-  duration_ms INTEGER,
-  request_id TEXT NOT NULL,
-  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
-);
-```
-
-Retain 30 days by default. Never store tool arguments or provider results in this table.
-
-## API contracts
-
-### `GET /integrations`
-
-Returns connection lifecycle, operational evidence, capabilities, billing summary, and control-plane release metadata. It does not probe providers.
-
-### `POST /integrations/:id/test`
-
-Runs the provider contract's safe diagnostic under normal tenant ownership and usage policy.
-
-Response:
-
-```json
-{
-  "ok": true,
-  "connectionId": "...",
-  "lifecycleStatus": "connected",
-  "operationalStatus": "available",
-  "checkedAt": "...",
-  "probe": {
-    "tool": "list-sites",
-    "summary": "Search Console returned 4 accessible properties"
-  },
-  "usage": {},
-  "requestId": "..."
-}
-```
-
-The summary is bounded and contains no unnecessary provider data. The operation follows the released connected-usage billing contract. The UI discloses that testing performs one provider operation.
-
-### `POST /integrations/:id/read`
-
-This is the renamed public facade for the existing generic bridge. The control plane validates tenant ownership, lifecycle, provider policy, live discovery, and arguments before execution.
-
-### `DELETE /integrations/:id`
-
-Requires a client idempotency key. Returns a deletion receipt. Stripe reconciliation happens in the main API only after the control plane confirms upstream and local deletion.
-
-### Internal control-plane endpoints
-
-Internal routes move from `/api/internal/nango/*` to `/api/internal/integrations/*`. Old routes remain aliases during cutover.
-
-Required internal groups:
-
-- catalog and provider-contract metadata;
-- connections list/get;
-- connect/reconnect session;
-- test/health;
-- describe/read/action;
-- export page;
-- action switch;
-- delete;
-- schedule bindings;
-- readiness and release metadata.
-
-## Frontend contract
-
-### Connection card states
-
-| Lifecycle | Operational | Primary badge | Secondary text | Primary recovery action |
-|---|---|---|---|---|
-| connected | available | Available | Last successful check timestamp | Test again |
-| connected | unknown | Connected | Not tested recently | Test connection |
-| connected | temporarily_unavailable | Temporarily unavailable | Safe error and timestamp | Retry test |
-| connected | rate_limited | Rate limited | Retry-after when present | Retry later |
-| connected | permission_limited | Limited access | Missing permissions/features | Refresh access |
-| reauth_required | any | Reconnect required | Authorization expired or revoked | Reconnect |
-| pending_oauth | unknown | Authorization pending | Finish OAuth | Continue authorization |
-| deleting | any | Disconnecting | Safe progress state | Retry deletion when failed |
-
-The phrase `Ready for sync` is reserved for a provider that:
-
-- is lifecycle connected;
-- has recent operational status available;
-- exposes every required sync tool;
-- has a successful sync policy validation.
-
-### Drawer controls
-
-Every connection drawer exposes:
-
-- Test connection;
-- Refresh access or Reconnect, with wording driven by lifecycle state;
-- Disconnect;
-- action switch when actions exist;
-- last successful call;
-- last failure and safe error code;
-- connection reference;
-- provider-contract version/hash in an operator details disclosure.
-
-Disconnect is not conditioned on transport type. It always requires a confirmation describing schedule impact and billing removal.
-
-### Failure messaging for this incident
-
-For `connection_transport_unavailable`, show:
-
-> Your Google authorization is still connected, but MCP Scraper could not reach the connected-service gateway. Retrying OAuth is unlikely to help. Try Test connection again later or contact support with request ID …
-
-Do not show `Reconnect required` unless lifecycle evidence supports it.
-
-## Root MCP contract
-
-Keep these tool names stable:
-
-- `list_service_connections`;
-- `describe_service_connection_tool`;
-- `read_service_connection`;
-- `call_service_connection_action`;
-- `export_connected_service_data`.
-
-Add:
-
-- `test_service_connection` for explicit bounded diagnostics.
-
-`list_service_connections` descriptions must state that `lifecycleStatus` and `operationalStatus` are different. `read_service_connection` and export errors must return the unified safe code, retryability, source layer, and request ID.
-
-The MCP instructions must stop calling a stored connection `healthy` merely because `reconnectRequired` is false.
-
-## Search Console analysis contract
-
-A comprehensive Search Console request follows this deterministic opening sequence:
-
-1. list tenant connections;
-2. select the exact Search Console connection;
-3. inspect lifecycle and operational evidence;
-4. if unknown or stale, call `test_service_connection`;
-5. call `list-sites` and enumerate accessible properties;
-6. select the requested property by normalized site URL/domain, never by brand-name guess alone;
-7. export or query a declared date range;
-8. validate row coverage and provider warnings;
-9. enrich only the relevant pages/queries with URL extraction, SERP, PAA, Maps, or competitors;
-10. distinguish Search Console facts from public-web inferences in the final report.
-
-If steps 4-7 fail, the analysis is blocked rather than silently replaced with public SERP data.
-
-## Reliability behavior
-
-### Deadlines
-
-- catalog/list: 10 seconds;
-- explicit test/read/describe: 20 seconds, including retries;
-- export page: 30 seconds;
-- aggregate export: existing bounded server orchestration deadline with continuation;
-- actions: provider-specific, with idempotency and outcome-unknown protection.
-
-### Circuit breaker
-
-Track control-plane/Nango transport failures separately from tenant credential failures.
-
-Open a short circuit when a deployment-wide threshold is met, for example five retryable transport failures across at least three tenants in one minute. While open:
-
-- fail fast with `connection_transport_unavailable`;
-- do not mark individual credentials for reconnect;
-- keep scheduled occurrences retryable without double billing;
-- surface one operator incident rather than many customer-auth incidents.
-
-### Scheduled occurrence behavior
-
-If the control plane is temporarily unavailable before provider execution begins:
-
-- do not consume the fixed scheduled-run charge;
-- release or retry the occurrence using its existing lease/idempotency identity;
-- retain checkpoint state;
-- emit a retryable infrastructure receipt.
-
-If provider execution has begun, use the existing billing and partial-result contracts.
-
-## Security and privacy
-
-1. Main API derives identity from authenticated context; browsers cannot choose another identity.
-2. Internal requests are signed, time bounded, and replay protected.
-3. Connection IDs are routing identifiers, not authorization.
-4. Foreign and absent connection IDs return the same public not-found response.
-5. Nango secret and connection routing IDs remain server-only.
-6. Logs contain request IDs, safe codes, provider keys, and durations—not credentials, query payloads, email contents, Search Console rows, or prompts.
-7. Health events contain no provider result bodies.
-8. Disconnect deletes credential routing and cascades bound data according to the published data-deletion policy.
-9. Actions remain gated by tenant ownership, live discovery, source policy, action switch, exact scheduled grant, and idempotency.
-
-## Observability
-
-### Required request context
-
-Propagate one request ID through:
-
-```text
-root MCP or dashboard
-  -> main API
-  -> integration control plane
-  -> Nango client/Proxy/function
-  -> billing receipt
-  -> health event
-```
-
-### Metrics
-
-Record by provider and operation class:
-
-- request count and latency;
-- success/error count;
-- safe error code;
-- retry count;
-- circuit state;
-- live discovery duration;
-- last successful provider operation;
-- lifecycle transitions;
-- deletion receipts pending/failed;
-- provider-contract hash mismatches;
-- main/control/scheduler release mismatch.
-
-### Alerts
-
-Alert operators when:
-
-- control-plane readiness fails;
-- Nango discovery failure rate exceeds threshold;
-- `connection_transport_unavailable` affects multiple tenants;
-- provider-contract hashes differ between deployments;
-- deletion receipts remain incomplete;
-- billing reconciliation differs from active Nango quantity;
-- a deployment serves a frontend bundle without the server's required controls.
-
-## Health endpoints
-
-The control-plane deployment exposes internal authenticated endpoints:
-
-```text
-GET /api/internal/integrations/health/live
-GET /api/internal/integrations/health/ready
-GET /api/internal/integrations/release
-```
-
-`live` proves the process can answer. `ready` verifies required configuration and bounded dependencies without enumerating customer connections:
-
-- Postgres connectivity;
-- Nango configuration present;
-- provider-contract load/hash;
-- main billing callback configuration;
-- schema migration compatibility.
-
-Nango network availability may be reported as a dependency state without making readiness permanently fail during a short upstream incident. The response distinguishes `ready`, `degraded`, and `not_ready`.
-
-## Implementation plan
-
-### Phase 0 — production repair and truthful UI
-
-Purpose: restore current functionality before architectural extraction.
-
-1. Inspect scheduler production logs for the failed request IDs.
-2. Verify `NANGO_SECRET_KEY`, `NANGO_MCP_URL`, `SCHEDULE_INTEGRATIONS_SECRET`, Postgres, and billing callback configuration in the scheduler deployment without exposing values.
-3. Call scheduler readiness and a sanitized Nango discovery smoke.
-4. Restore the existing transport and prove Search Console `list-sites`.
-5. Deploy the current main frontend bundle so Nango connections show Disconnect.
-6. Replace `Connected · Ready for sync` with lifecycle-only wording until operational health fields ship.
-7. Add a production smoke that opens the Search Console drawer and asserts Refresh/Test/Disconnect controls.
-
-Exit gate:
-
-- current connection lists Search Console sites;
-- bounded export returns rows or a valid empty/provider-warning result;
-- live UI exposes Disconnect;
-- no customer credential is deleted or recreated solely to repair the transport.
-
-### Phase 1 — extract the shared integration module
-
-Move connection-control code in `mcp-scraper-scheduler` behind stable internal interfaces:
-
-```text
-src/mastra/integrations/contracts/
-src/mastra/integrations/repository/
-src/mastra/integrations/execution/
-src/mastra/integrations/health/
-src/mastra/integrations/http/
-src/mastra/integrations/billing/
-```
-
-Keep compatibility re-exports while scheduler imports migrate.
-
-Exit gate:
-
-- scheduler tests pass using the extracted module;
-- direct read/action/export route tests pass without importing schedule execution code;
-- one canonical provider contract generates all projections.
-
-### Phase 2 — deploy the independent control plane
-
-1. Create the separate Vercel project and domain.
-2. Configure its Postgres, Nango, internal signing, billing callback, and runtime settings.
-3. Apply additive schema migrations.
-4. Deploy internal route aliases and health/release endpoints.
-5. Run sanitized catalog, list, test, read, describe, export-page, reconnect-session creation, and non-destructive deletion-fixture tests in staging.
-
-Exit gate:
-
-- independent readiness is green;
-- its provider-contract hash matches the scheduler build;
-- it can read a staging Search Console fixture without the scheduler deployment.
-
-### Phase 3 — main API shadow and cutover
-
-Add environment variables:
-
-```text
-INTEGRATIONS_CONTROL_URL
-INTEGRATIONS_CONTROL_KEY_CURRENT
-INTEGRATIONS_CONTROL_KEY_PREVIOUS
-INTEGRATIONS_CONTROL_MODE=legacy|shadow|primary
-```
-
-Modes:
-
-- `legacy`: use `NANGO_CONTROL_URL` scheduler path;
-- `shadow`: execute safe catalog/list/health comparisons against both; only legacy result is customer-visible; never shadow actions or deletes;
-- `primary`: use the independent control plane, retain legacy fallback only for explicitly allowed read-only operations during the rollback window.
-
-Do not automatically fail over actions, connect/reconnect sessions, or deletion across services because duplicate side effects are possible.
-
-Exit gate:
-
-- shadow comparisons meet parity thresholds;
-- Search Console list/read/export works with scheduler deployment disabled in staging;
-- request IDs and usage receipts reconcile.
-
-### Phase 4 — health state and frontend
-
-1. Ship additive health columns and event repository.
-2. Record health on every execution path.
-3. Add public health fields and `test_service_connection`.
-4. update frontend states, timestamps, controls, and failure guidance;
-5. update MCP schemas, descriptions, generated manifest, MCPB, docs, and SDK types.
-
-Exit gate:
-
-- a simulated control-plane 503 shows `Connected · Temporarily unavailable`;
-- it does not show Reconnect required;
-- a successful test transitions to Available;
-- a confirmed revoked grant transitions to Reconnect required.
-
-### Phase 5 — scheduler cutover and cleanup
-
-1. Make scheduled paths consume the extracted shared integration module.
-2. Remove direct public-operation responsibility from the scheduler deployment.
-3. Keep old internal route aliases for one release window with deprecation metrics.
-4. Remove the main API default fallback to `mcp-scraper-scheduler.vercel.app` after production proof.
-5. Rename public code concepts from `scheduleConnection*` to `serviceConnection*` or `integration*`, retaining compatibility exports where required.
-
-Exit gate:
-
-- stopping the scheduler web deployment does not break direct provider reads;
-- stopping the control plane blocks direct and scheduled provider execution with the same safe operational error, without corrupting schedule state;
-- no old route traffic remains before alias removal.
-
-## Repository task breakdown
-
-### `mcp-scraper-scheduler`
-
-- IC-01: Extract canonical provider contracts and generated manifest.
-- IC-02: Extract tenant connection repository and lifecycle state transitions.
-- IC-03: Extract provider executor for discovery/read/action/export.
-- IC-04: Add operational-health repository and events.
-- IC-05: Add idempotent deletion workflow and receipts.
-- IC-06: Add signed internal HTTP verification.
-- IC-07: Add independent control-plane entrypoint and health/release routes.
-- IC-08: Migrate scheduled agent and deterministic sync to shared executor.
-- IC-09: Add circuit breaker, retry policy, and safe error mapping.
-- IC-10: Add staging fixtures and integration contract tests.
-
-### `mcp-scraper`
-
-- IC-11: Add integration-control client with legacy/shadow/primary modes.
-- IC-12: Add `/integrations` routes and `/schedule-connections` aliases.
-- IC-13: Extend connection schemas with lifecycle and operational health.
-- IC-14: Add `test_service_connection` MCP tool.
-- IC-15: Update Search Console tool guidance and analysis opening sequence.
-- IC-16: Make Disconnect transport-neutral and add Test connection UI.
-- IC-17: Replace optimistic readiness badges with evidence-backed states.
-- IC-18: Add release/contract-hash parity checks.
-- IC-19: Update generated tool manifests, MCPB, docs, legal deletion wording, and SDK-facing schemas.
-- IC-20: Add production smoke and rollback verification.
-
-## Test plan
-
-### Unit tests
-
-- lifecycle-to-operational state mapping;
-- safe error taxonomy;
-- provider-contract generation and hash stability;
-- Search Console safe probe definition;
-- health TTL expiry;
-- retry eligibility and deadline budget;
-- circuit-breaker transitions;
-- disconnect idempotency;
-- reconnect preserves logical connection identity;
-- billing quantity only changes after confirmed deletion;
-- HMAC request signing, expiry, replay rejection, and key rotation.
-
-### Contract tests
-
-- main and control-plane request/response schemas;
-- old and new route aliases are equivalent;
-- root MCP schema includes health fields and request IDs;
-- frontend bundle contains Test, Refresh/Reconnect, and Disconnect for Nango;
-- frontend does not use `reconnectRequired === false` as readiness;
-- provider-contract hashes match across repos/deployments;
-- generated bundle and JSX share the same connection controls;
-- health events never persist arguments or provider bodies.
-
-### Integration tests
-
-- active connection plus unavailable control plane;
-- invalid internal signature;
-- Nango MCP discovery 503;
-- provider 401/revoked grant;
-- provider 403/missing permission;
-- provider 429 with Retry-After;
-- successful Search Console `list-sites` with zero and multiple properties;
-- Search Analytics valid empty result;
-- aggregate export continuation;
-- delete with Nango success and billing reconcile success;
-- delete with Nango failure retains local state and billing quantity;
-- reconnect does not create duplicate billing quantity;
-- schedule binding survives reconnect;
-- schedule binding cascades on confirmed disconnect.
-
-### End-to-end staging tests
-
-1. Disable scheduler deployment; direct Search Console test/read/export still passes.
-2. Disable integration control plane; direct calls fail safely and UI shows temporary unavailability.
-3. Restore control plane; Test connection changes operational state to available.
-4. Revoke a staging Google grant; lifecycle becomes reauth required.
-5. Reconnect; the same logical connection and bindings become available.
-6. Disconnect a disposable fixture; Nango credential, local row, bindings, synchronized data, and recurring quantity are removed exactly once.
-7. Run a scheduled Search Console sync through the shared executor.
-8. Verify all request, health, usage, and schedule receipts share traceable IDs.
-
-### Production proof
-
-For the existing Search Console tenant, collect only sanitized evidence:
-
-- main/control/scheduler release IDs;
-- provider-contract hashes;
-- connection lifecycle and operational status;
-- successful `list-sites` count and matched Geny's Flowers property URL;
-- bounded export count/date range/warnings;
-- UI screenshot showing Test, Refresh access, and Disconnect;
-- one request trace across main and control plane;
-- billing quantity before and after no destructive action.
-
-Do not include queries, page rows, tokens, credentials, or private property data in release logs.
-
-## Acceptance criteria
-
-The fix is complete only when all of the following are true:
-
-1. A one-time connected read does not require the scheduler deployment.
-2. Scheduler downtime does not break direct Search Console reads.
-3. Control-plane downtime produces `connection_transport_unavailable` with request ID and retryability.
-4. That downtime does not mark credentials for reconnect.
-5. The UI distinguishes lifecycle and operational health.
-6. `Connected` alone is never rendered as `Ready for sync`.
-7. Search Console Test connection performs the bounded `list-sites` diagnostic.
-8. The Geny's Flowers workflow selects an exact accessible property before querying performance.
-9. Refresh/Reconnect and Disconnect are visible for Nango connections.
-10. Disconnect removes upstream credential routing, local data, bindings, and billable quantity exactly once.
-11. Reconnect preserves the logical connection and billing quantity.
-12. Main API, control plane, and scheduler use the same provider-contract hash.
-13. Direct and scheduled calls use the same execution, safety, and error-normalization code.
-14. Usage receipts remain idempotent and the main Credit ledger remains authoritative.
-15. Production bundle and backend release parity are verified automatically.
-16. Existing healthy connections require no migration-time reauthorization.
-17. All unit, contract, integration, staging, and production-smoke gates pass.
-
-## Rollback
-
-Rollback is controlled by `INTEGRATIONS_CONTROL_MODE`:
-
-1. return main API to `legacy` for read-only direct operations;
-2. keep the new additive database columns and event tables;
-3. stop new control-plane traffic without deleting its deployment;
-4. do not roll back a completed disconnect or reconnect by replaying it elsewhere;
-5. keep scheduler on the shared module if its tests passed—code extraction is not coupled to traffic cutover;
-6. retain release/hash evidence and failed request IDs for diagnosis.
-
-Rollback is not allowed to restore the misleading UI. Even in legacy mode, lifecycle and operational status remain separate and transport failures remain temporary-unavailability errors.
-
-## Risks and mitigations
-
-| Risk | Mitigation |
-|---|---|
-| Shared Postgres access from two deployments | One migration owner, additive migrations, repository-level transactions, readiness schema check |
-| Duplicate provider calls during shadowing | Shadow only catalog/list/health metadata; never shadow actions, OAuth sessions, deletes, or billed exports |
-| New deployment adds latency | Same-region deployment, keep-alive, bounded timeouts, trace metrics |
-| HMAC key rotation outage | Current and previous key acceptance with short overlap and explicit key IDs |
-| Health probe consumes provider usage | No page-load probes; explicit test discloses one provider operation; connect probe is bounded |
-| Operational status becomes stale | Display timestamp and TTL; expire to unknown, never assume healthy |
-| Provider policy drift during migration | Canonical generated contract and release hash gate |
-| Deletion partially completes | Durable idempotent receipt, ordered upstream-first deletion, retryable `deleting` state |
-| Existing schedules bind old IDs | Reconnect preserves logical ID; disconnect cascades intentionally; migration keeps existing rows |
-| Production frontend remains stale | Bundle/source contract test plus live release-ID smoke |
-
-## Evidence anchors
-
-Current source locations supporting this specification:
-
-- `src/api/nango-control.ts`: main-to-scheduler control URL, connection facade, read/action/export forwarding, error normalization.
-- `src/api/server.ts`: public connection, reconnect, delete, read, describe, export, action, and schedule-binding routes.
-- `src/mcp/http-mcp-tool-executor.ts`: root MCP-to-main connection bridge.
-- `public/app.jsx` and `public/app.js`: connection drawer controls and current local transport-neutral Disconnect implementation.
-- `tests/contract/integrations-ui.test.ts`: existing UI/API route contract checks.
-- `tests/unit/nango-control.test.ts`: current main-to-control request contract.
-- `mcp-scraper-scheduler/src/mastra/routes/nango-integrations.ts`: internal Nango catalog, connection lifecycle, read/action/describe/export routes.
-- `mcp-scraper-scheduler/src/mastra/db/external-connections.ts`: tenant connection rows, bindings, checkpoints, synchronized records, run receipts, and cascade relationships.
-- `mcp-scraper-scheduler/src/mastra/lib/nango-connection-client.ts`: Nango MCP credential boundary and per-connection client.
-- `mcp-scraper-scheduler/src/mastra/lib/nango-api.ts`: canonical candidate provider catalog and policy source to consolidate.
-- `mcp-scraper-scheduler/src/mastra/lib/connected-usage-billing.ts`: connected usage authorization and settlement boundary.
-
-## Final product statement
-
-A connected account and an available connection are not the same fact.
-
-MCP Scraper will store OAuth lifecycle truth, measure operational truth, present both honestly, and execute direct provider work through a dedicated integration control plane. Scheduling remains an optional consumer of that capability rather than a hidden prerequisite for every connected-service call.