mcp-scraper 0.25.0 → 0.26.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/README.md +3 -3
  2. package/dist/bin/api-server.cjs +4980 -2927
  3. package/dist/bin/api-server.cjs.map +1 -1
  4. package/dist/bin/api-server.js +2 -2
  5. package/dist/bin/mcp-scraper-cli.cjs +1 -1
  6. package/dist/bin/mcp-scraper-cli.cjs.map +1 -1
  7. package/dist/bin/mcp-scraper-cli.js +1 -1
  8. package/dist/bin/mcp-scraper-install.cjs +2 -2
  9. package/dist/bin/mcp-scraper-install.cjs.map +1 -1
  10. package/dist/bin/mcp-scraper-install.js +2 -2
  11. package/dist/bin/mcp-stdio-server.cjs +52 -4
  12. package/dist/bin/mcp-stdio-server.cjs.map +1 -1
  13. package/dist/bin/mcp-stdio-server.js +4 -4
  14. package/dist/{chunk-PNBYYAEK.js → chunk-3ZZOLQ6A.js} +2 -2
  15. package/dist/{chunk-JHI373VB.js → chunk-4767BA2O.js} +85 -3
  16. package/dist/chunk-4767BA2O.js.map +1 -0
  17. package/dist/chunk-FQQC5OZJ.js +7 -0
  18. package/dist/chunk-FQQC5OZJ.js.map +1 -0
  19. package/dist/{chunk-YNHPZU6B.js → chunk-TBRX7VOP.js} +2 -2
  20. package/dist/{chunk-YNHPZU6B.js.map → chunk-TBRX7VOP.js.map} +1 -1
  21. package/dist/{chunk-NC37L5RW.js → chunk-YMEPX46N.js} +47 -5
  22. package/dist/chunk-YMEPX46N.js.map +1 -0
  23. package/dist/{server-PILR3U5E.js → server-UHDJ5TH5.js} +2484 -580
  24. package/dist/server-UHDJ5TH5.js.map +1 -0
  25. package/dist/{site-extract-repository-2QQSL2QC.js → site-extract-repository-O36SIFI2.js} +3 -3
  26. package/dist/{worker-RV2TYXE3.js → worker-E3KHQH2F.js} +2 -2
  27. package/docs/mcp-tool-manifest.generated.json +164 -5
  28. package/docs/specs/main-mcp-integration-ownership-spec.md +1164 -0
  29. package/package.json +5 -1
  30. package/dist/chunk-JHI373VB.js.map +0 -1
  31. package/dist/chunk-NC37L5RW.js.map +0 -1
  32. package/dist/chunk-UELJZCUC.js +0 -7
  33. package/dist/chunk-UELJZCUC.js.map +0 -1
  34. package/dist/server-PILR3U5E.js.map +0 -1
  35. /package/dist/{chunk-PNBYYAEK.js.map → chunk-3ZZOLQ6A.js.map} +0 -0
  36. /package/dist/{site-extract-repository-2QQSL2QC.js.map → site-extract-repository-O36SIFI2.js.map} +0 -0
  37. /package/dist/{worker-RV2TYXE3.js.map → worker-E3KHQH2F.js.map} +0 -0
@@ -0,0 +1,1164 @@
1
+ # Main MCP Integration Ownership
2
+
3
+ - Status: Implementation in progress — Nango ownership cutover implemented locally; Resend credential migration and production rollout remain
4
+ - Spec date: 2026-07-16
5
+ - Product decision: Integrations belong to the main MCP Scraper system
6
+ - Scheduler decision: Scheduling remains an independent Mastra system and consumes main-MCP integration APIs
7
+ - Primary repos: `mcp-scraper`, `mcp-scraper-scheduler`
8
+ - Incident driver: Search Console displayed `Connected` while direct reads and export returned `connection_transport_unavailable`
9
+ - Supersedes: `connected-services-control-plane-decoupling-spec.md`
10
+ - Preserves: the released connected-account and usage-pricing contract in `unified-credit-and-scheduled-execution-billing-spec.md`
11
+
12
+ ## Implementation receipt — 2026-07-16
13
+
14
+ Implemented in the working tree:
15
+
16
+ - main-owned Turso connection, health-event, action-audit, and scheduler-nonce tables;
17
+ - stable public IDs reconciled from server-only Nango routing IDs;
18
+ - direct main-MCP Nango REST and MCP SDK transport for discovery, connect, reconnect, reads, actions, and Search Console export;
19
+ - local connected-usage preflight and idempotent settlement;
20
+ - signed scheduler describe/execute APIs with expiry and replay protection;
21
+ - Mastra proxy tools and deterministic sync execution through main;
22
+ - scheduler logical bindings with no connection foreign key;
23
+ - idempotent scheduler purge endpoint used by disconnect;
24
+ - lifecycle and operational-health separation in the frontend and MCP schemas;
25
+ - `test_service_connection` plus the dashboard Test connection control;
26
+ - canonical `/integrations` aliases while preserving `/schedule-connections` compatibility routes.
27
+
28
+ Still required before the full specification is complete:
29
+
30
+ - migrate the Resend remote-MCP credential and OAuth lifecycle from scheduler Postgres into encrypted main-MCP storage;
31
+ - migrate existing scheduler Nango rows and bindings in production while preserving public IDs;
32
+ - deploy both repos with the shared signing secret and `INTEGRATION_OWNER=main`;
33
+ - remove the rollback-only Nango routes, secrets, and legacy tables after the observation window.
34
+
35
+ ## Executive decision
36
+
37
+ The main MCP Scraper system will own integrations end to end.
38
+
39
+ This includes:
40
+
41
+ - the tenant-owned connection record;
42
+ - Nango and remote-MCP credential routing;
43
+ - OAuth connect and reconnect flows;
44
+ - provider catalog and source-controlled tool policy;
45
+ - live schema and capability discovery;
46
+ - direct reads and gated actions;
47
+ - bulk export page execution;
48
+ - operational health and lifecycle state;
49
+ - disconnect and credential deletion;
50
+ - connection audit and deletion receipts;
51
+ - connected-usage authorization and settlement;
52
+ - the public REST, frontend, SDK, and root MCP contracts.
53
+
54
+ The Mastra scheduler remains a separate system. It owns schedules, due occurrences, leases, agent execution, deterministic sync orchestration, checkpoints, synchronized records, and scheduled-run receipts. When scheduled work needs a connected provider, the scheduler calls the main MCP through signed internal integration APIs. It does not hold Nango secrets, provider credentials, or an authoritative copy of the connection record.
55
+
56
+ There will be no third integration service.
57
+
58
+ ```text
59
+ Direct use
60
+ User / Codex / Claude
61
+ -> Main MCP Scraper
62
+ -> tenant connection repository
63
+ -> Nango or remote MCP
64
+ -> provider
65
+
66
+ Scheduled use
67
+ Inngest -> Mastra scheduler
68
+ -> signed internal call to Main MCP Scraper
69
+ -> tenant connection repository
70
+ -> Nango or remote MCP
71
+ -> provider
72
+ -> scheduler checkpoint / synchronized records / run receipt
73
+ ```
74
+
75
+ The scheduler is a consumer of integrations, not their owner or transport.
76
+
77
+ ## Product contract in plain language
78
+
79
+ ### Immediate provider use
80
+
81
+ When a user asks Codex to analyze Search Console, the main MCP directly resolves the user's Search Console connection, checks its state and allowed tools, calls Nango, and returns the provider result. No schedule, Mastra workflow, Inngest occurrence, or scheduler deployment is involved.
82
+
83
+ ### Scheduled provider use
84
+
85
+ When a scheduled action becomes due, the scheduler claims the occurrence and invokes the same main-MCP integration capability through a signed internal request. Mastra may reason over proxied provider tools, while deterministic sync uses a bounded sync-slice API. The main MCP still owns authorization, provider execution, action gates, and connected-usage billing.
86
+
87
+ ### Connection state
88
+
89
+ `Connected` means that the OAuth or remote-MCP credential lifecycle is active. It does not promise that Nango or the provider is currently reachable. The product displays credential lifecycle and operational availability separately.
90
+
91
+ ### Recovery
92
+
93
+ Transport outages produce `Temporarily unavailable`, not `Reconnect required`. Reconnect appears only when authorization evidence proves that the credential expired, was revoked, or lacks a required scope.
94
+
95
+ ## Why the current system is wrong
96
+
97
+ ### Current direct path
98
+
99
+ The public main API exposes connection routes and root MCP bridge tools, but `src/api/nango-control.ts` defaults its internal target to:
100
+
101
+ ```text
102
+ https://mcp-scraper-scheduler.vercel.app
103
+ ```
104
+
105
+ Connect, reconnect, disconnect, list, describe, read, action, binding, and export-page operations are forwarded into scheduler routes. A normal direct Search Console call therefore depends on a deployment whose product purpose is scheduled work.
106
+
107
+ ### Current state ownership
108
+
109
+ The scheduler Postgres database currently contains:
110
+
111
+ - `mem_external_connections`;
112
+ - remote-MCP credential records;
113
+ - action audit;
114
+ - schedule-to-connection bindings;
115
+ - connection sync checkpoints and records;
116
+ - connection sync run receipts.
117
+
118
+ The first three are integration-owned data. Bindings, checkpoints, synchronized records, and scheduled receipts are scheduler-owned data. They must be separated.
119
+
120
+ ### Current execution ownership
121
+
122
+ The scheduler repository currently contains Nango and remote-MCP connection clients, live tool discovery, provider policies, direct reads/actions, exports, OAuth sessions, lifecycle changes, and usage metering. Mastra schedules use these directly, but the main MCP also proxies through them.
123
+
124
+ This makes the scheduler an accidental integration backend.
125
+
126
+ ### Current UI truth gap
127
+
128
+ The live frontend presented Search Console as `Connected · Ready for sync` because a row existed and approved tools were listed. Actual provider calls returned repeatable HTTP 503 transport failures. The deployed frontend also hid Disconnect for Nango connections despite current local source supporting it.
129
+
130
+ ## Goals
131
+
132
+ 1. Make the main MCP the sole integration authority.
133
+ 2. Make direct connected-provider calls independent of scheduler availability.
134
+ 3. Keep the scheduler as an autonomous Mastra/Inngest system.
135
+ 4. Remove Nango and remote-MCP secrets from the scheduler.
136
+ 5. Move connection lifecycle, health, policy, audit, and connected-usage settlement to the main MCP.
137
+ 6. Preserve public connection IDs and existing healthy credentials during migration.
138
+ 7. Preserve schedules, bindings, checkpoints, synchronized records, and run history.
139
+ 8. Separate credential lifecycle from operational health in every public surface.
140
+ 9. Keep action safety and connected-usage billing identical for direct and scheduled calls.
141
+ 10. Provide a reversible migration with explicit proof and no forced customer reconnection.
142
+
143
+ ## Non-goals
144
+
145
+ - Merging the scheduler repository into the main MCP repository.
146
+ - Moving Mastra agents, Inngest functions, schedule leases, or sync checkpoints into the main MCP.
147
+ - Replacing Nango as the OAuth/token-refresh provider.
148
+ - Changing connected-account pricing or Credit rates.
149
+ - Replacing deterministic scheduled sync with an LLM.
150
+ - Moving synchronized provider datasets from scheduler Postgres to Turso.
151
+ - Creating a third integration deployment.
152
+ - Renaming every legacy route in the first release.
153
+ - Requiring users to reconnect credentials that can be migrated safely.
154
+
155
+ ## Ownership after migration
156
+
157
+ | Capability or data | Main MCP | Mastra scheduler | Nango / remote provider |
158
+ |---|---:|---:|---:|
159
+ | Public user/API-key authentication | Yes | No | No |
160
+ | User and billing identity | Yes | No | No |
161
+ | Public connection ID | Yes | Reference only | No |
162
+ | Connection lifecycle record | Yes | No | No |
163
+ | Nango credential/token | Routing metadata only | No | Yes |
164
+ | Remote-MCP encrypted credential | Yes | No | Provider may also hold OAuth grant |
165
+ | Provider read/action policy | Canonical | Generated consumer projection | No |
166
+ | Live tool discovery | Yes | No | Supplies inventory |
167
+ | Direct read/action/export | Yes | No | Executes provider work |
168
+ | Operational health | Yes | Receives response only | Supplies evidence |
169
+ | Connected-usage debit | Yes, authoritative | No | Supplies usage measurement |
170
+ | Schedule definition and occurrence | No | Yes | No |
171
+ | Schedule-to-connection grant | Scheduler stores; main validates signed assertion | Yes | No |
172
+ | Mastra tool proxy | Executes endpoint | Builds proxy tool | No |
173
+ | Deterministic sync checkpoint | No | Yes | No |
174
+ | Synchronized records/table | Pointer only when useful | Yes | No |
175
+ | Scheduled-run base/model billing | Authoritative ledger | Authorizes and reports | Model provider supplies cost |
176
+
177
+ ## Architectural decisions
178
+
179
+ ### D1. Main MCP owns the connection record in Turso
180
+
181
+ Create main-owned tables in the existing Turso/libSQL database. The authoritative owner is the stable `users.id`, not an email string.
182
+
183
+ ```sql
184
+ CREATE TABLE service_connections (
185
+ id TEXT PRIMARY KEY,
186
+ user_id INTEGER NOT NULL REFERENCES users(id),
187
+ provider_config_key TEXT NOT NULL,
188
+ provider TEXT,
189
+ transport TEXT NOT NULL,
190
+ upstream_connection_id TEXT,
191
+ label TEXT,
192
+ lifecycle_status TEXT NOT NULL,
193
+ operational_status TEXT NOT NULL DEFAULT 'unknown',
194
+ reconnect_required INTEGER NOT NULL DEFAULT 0,
195
+ actions_enabled INTEGER NOT NULL DEFAULT 0,
196
+ granted_permissions_json TEXT,
197
+ enabled_features_json TEXT,
198
+ schema_revision TEXT,
199
+ provider_contract_version TEXT NOT NULL,
200
+ provider_contract_hash TEXT NOT NULL,
201
+ last_checked_at TEXT,
202
+ last_successful_call_at TEXT,
203
+ last_failure_at TEXT,
204
+ last_failure_code TEXT,
205
+ last_failure_retryable INTEGER,
206
+ health_evidence_source TEXT,
207
+ consecutive_failures INTEGER NOT NULL DEFAULT 0,
208
+ scheduler_table_name TEXT,
209
+ scheduler_vault_name TEXT,
210
+ legacy_owner_identity TEXT,
211
+ created_at TEXT NOT NULL DEFAULT (datetime('now')),
212
+ updated_at TEXT NOT NULL DEFAULT (datetime('now')),
213
+ UNIQUE(user_id, provider_config_key, upstream_connection_id)
214
+ );
215
+ ```
216
+
217
+ `upstream_connection_id` is server-only and is never returned to browsers or MCP clients. The existing public UUID remains `id`.
218
+
219
+ ### D2. Main MCP owns remote-MCP credentials encrypted at rest
220
+
221
+ Nango continues to store Nango-backed OAuth credentials. The main MCP stores only Nango routing metadata.
222
+
223
+ For remote MCPs such as Resend, create an encrypted credential table:
224
+
225
+ ```sql
226
+ CREATE TABLE service_connection_credentials (
227
+ connection_id TEXT PRIMARY KEY REFERENCES service_connections(id) ON DELETE CASCADE,
228
+ credential_kind TEXT NOT NULL,
229
+ endpoint TEXT NOT NULL,
230
+ ciphertext BLOB NOT NULL,
231
+ nonce BLOB NOT NULL,
232
+ auth_tag BLOB NOT NULL,
233
+ key_version TEXT NOT NULL,
234
+ oauth_client_metadata_json TEXT,
235
+ created_at TEXT NOT NULL DEFAULT (datetime('now')),
236
+ updated_at TEXT NOT NULL DEFAULT (datetime('now'))
237
+ );
238
+ ```
239
+
240
+ Use envelope encryption with a versioned production secret or KMS-backed data key. Never store plaintext API keys, access tokens, refresh tokens, or OAuth client secrets in Turso. Key rotation must support current and previous key versions.
241
+
242
+ ### D3. Main MCP uses the MCP SDK, not Mastra, for provider transport
243
+
244
+ The main repository already depends on `@modelcontextprotocol/sdk`. Move or rewrite the connection client using the standard MCP SDK transport. Do not add Mastra to the main MCP merely to call Nango's MCP endpoint or a remote MCP.
245
+
246
+ The executor interface is provider-neutral:
247
+
248
+ ```ts
249
+ interface IntegrationExecutor {
250
+ listTools(connection: OwnedServiceConnection): Promise<DiscoveredTool[]>
251
+ describeTool(connection: OwnedServiceConnection, tool: string): Promise<SanitizedTool>
252
+ read(connection: OwnedServiceConnection, tool: string, args: JsonObject, context: ExecutionContext): Promise<ExecutionResult>
253
+ action(connection: OwnedServiceConnection, tool: string, args: JsonObject, context: ActionContext): Promise<ExecutionResult>
254
+ exportPage(connection: OwnedServiceConnection, request: ExportPageRequest, context: ExecutionContext): Promise<ExportPageResult>
255
+ disconnect(connection: OwnedServiceConnection, context: MutationContext): Promise<DisconnectResult>
256
+ }
257
+ ```
258
+
259
+ Nango and remote-MCP transports implement the same interface.
260
+
261
+ ### D4. Provider policy is canonical in the main MCP
262
+
263
+ Move safe read/action classifications, administrative denylists, required permissions/features, sync requirements, health probes, and schema rules into:
264
+
265
+ ```text
266
+ mcp-scraper/src/integrations/contracts/
267
+ ```
268
+
269
+ Generate a scheduler-consumer artifact containing only what scheduled configuration and proxy-tool construction require. The artifact has a version and SHA-256 hash. Scheduler builds fail when their projection is stale.
270
+
271
+ Callable tools remain an intersection:
272
+
273
+ ```text
274
+ live upstream tools
275
+ intersect main-owned provider policy
276
+ intersect active tenant-owned connection
277
+ intersect verified provider permissions/features
278
+ intersect account action switch for mutations
279
+ intersect signed schedule grant for scheduled calls
280
+ ```
281
+
282
+ The scheduler cannot expand this set.
283
+
284
+ ### D5. Scheduler stores only logical connection references and grants
285
+
286
+ The scheduler continues to own `mem_scheduled_action_connections`, but the table becomes a reference to a main-owned public connection ID, not a foreign key to a scheduler-owned connection row.
287
+
288
+ Migration shape:
289
+
290
+ ```sql
291
+ ALTER TABLE mem_scheduled_action_connections
292
+ ADD COLUMN IF NOT EXISTS connection_public_id TEXT,
293
+ ADD COLUMN IF NOT EXISTS provider_config_key TEXT,
294
+ ADD COLUMN IF NOT EXISTS connection_label TEXT,
295
+ ADD COLUMN IF NOT EXISTS provider_contract_hash TEXT;
296
+ ```
297
+
298
+ Backfill `connection_public_id` from the existing `external_connection_id`. After cutover, remove the foreign key to `mem_external_connections` in a controlled table migration. The scheduler validates current ownership and availability through the main MCP before a binding is created or edited.
299
+
300
+ ### D6. Scheduler calls signed internal main-MCP integration endpoints
301
+
302
+ Add a scoped service-auth contract distinct from browser/API-key authentication and distinct from billing-only secrets.
303
+
304
+ ```text
305
+ INTERNAL_INTEGRATIONS_KEY_CURRENT
306
+ INTERNAL_INTEGRATIONS_KEY_PREVIOUS
307
+ MCP_SCRAPER_INTERNAL_URL
308
+ ```
309
+
310
+ Every request includes:
311
+
312
+ - scheduler service identity;
313
+ - issued-at and expiry;
314
+ - request ID;
315
+ - method and path;
316
+ - body hash;
317
+ - occurrence ID when scheduled;
318
+ - schedule ID;
319
+ - tenant identity assertion;
320
+ - connection public ID;
321
+ - exact granted tool set or one requested granted tool;
322
+ - provider-contract hash;
323
+ - HMAC signature.
324
+
325
+ The main MCP verifies the signature, expiry, replay nonce, tenant mapping, connection ownership, provider contract, requested tool, action switch, and billing authorization.
326
+
327
+ ### D7. Mastra receives proxy tools, never Nango credentials
328
+
329
+ Replace `scheduled-action-nango-tools.ts` direct Nango clients with main-MCP proxy tools. Mastra still sees provider-shaped tools with bounded schemas, but execution performs a signed call to the main MCP.
330
+
331
+ ```text
332
+ Mastra tool.execute(args)
333
+ -> internal main MCP /integrations/:id/read or /action
334
+ -> main-owned executor
335
+ -> Nango/remote MCP
336
+ -> normalized result and usage receipt
337
+ ```
338
+
339
+ The scheduler may cache sanitized tool schemas by `connectionId + schemaRevision + providerContractHash`. It never caches credentials or upstream connection IDs.
340
+
341
+ ### D8. Deterministic sync uses a bounded sync-slice endpoint
342
+
343
+ Scheduled `connection_sync` remains deterministic and scheduler-owned. It calls:
344
+
345
+ ```text
346
+ POST /api/internal/integrations/:id/sync-slice
347
+ ```
348
+
349
+ Input includes dataset, bounded time range, page size, cursor, schedule ID, and occurrence ID. Main validates and executes provider work. Scheduler persists normalized records, checkpoints, typed tables, and the run receipt.
350
+
351
+ Main returns usage receipts so connected function/Proxy/compute costs settle once in the authoritative main ledger. The scheduler charges the scheduled occurrence base separately under its existing contract.
352
+
353
+ ### D9. Direct connected usage settles locally in main
354
+
355
+ After migration, a direct integration call no longer makes a scheduler-to-main billing callback. Main authorizes and settles connected usage against its own Credit ledger in the same product boundary.
356
+
357
+ For scheduled integration calls, the signed request carries an occurrence-scoped idempotency identity. Main settles connected usage; scheduler settles only the scheduled base and model cost. Duplicate occurrence retries cannot duplicate connected usage debits.
358
+
359
+ ### D10. Lifecycle and operational health are separate
360
+
361
+ ```ts
362
+ type ConnectionLifecycleStatus =
363
+ | 'pending_oauth'
364
+ | 'connected'
365
+ | 'reauth_required'
366
+ | 'deleting'
367
+ | 'cleanup_pending'
368
+ | 'disconnected'
369
+
370
+ type ConnectionOperationalStatus =
371
+ | 'unknown'
372
+ | 'available'
373
+ | 'temporarily_unavailable'
374
+ | 'rate_limited'
375
+ | 'permission_limited'
376
+ | 'provider_error'
377
+ ```
378
+
379
+ Only confirmed authentication evidence changes lifecycle to `reauth_required`. Transport 5xx, timeout, Nango discovery failure, billing unavailability, and provider rate limiting update operational health without invalidating the credential.
380
+
381
+ ### D11. Health comes from actual operations and explicit tests
382
+
383
+ The Integrations page does not probe every provider on load.
384
+
385
+ Health evidence comes from:
386
+
387
+ - connect/reconnect verification;
388
+ - explicit `test_service_connection`;
389
+ - direct read/action/describe/export;
390
+ - scheduled proxy calls and sync slices.
391
+
392
+ Evidence expires to `unknown` after a provider-contract TTL. The UI always displays the last-check timestamp and safe error code when present.
393
+
394
+ For Search Console, the safe diagnostic is `list-sites` with empty arguments. Success proves the main MCP reached Nango, Google accepted the credential, and Search Console returned a valid site collection. It does not prove a named property exists or has performance rows.
395
+
396
+ ### D12. Disconnect is a distributed deletion saga
397
+
398
+ Main owns the deletion request and durable receipt:
399
+
400
+ 1. verify user ownership and idempotency key;
401
+ 2. mark lifecycle `deleting`;
402
+ 3. revoke/delete the upstream Nango or remote-MCP credential;
403
+ 4. call the scheduler's idempotent purge endpoint for bindings, checkpoints, sync data, and pending work tied to the public connection ID;
404
+ 5. delete main encrypted credentials and connection metadata;
405
+ 6. reconcile the recurring Nango quantity in Stripe;
406
+ 7. finalize the deletion receipt.
407
+
408
+ If step 3 fails, retain the active local record in `deleting` and retry safely. If step 4 fails after upstream revocation, set `cleanup_pending`; the connection is unusable and billing quantity is removed only when the defined commercial deletion point is satisfied. The cleanup worker retries without recreating credentials.
409
+
410
+ Ordinary users never receive a force-local-delete path. Administrative override is separately authorized and audited.
411
+
412
+ ### D13. Reconnect preserves the logical connection
413
+
414
+ Reconnect updates the same main-owned public connection ID. It preserves schedule bindings, table/vault pointers, audit history, billing identity, and health history. If Nango rotates the upstream routing ID, only the server-only field changes.
415
+
416
+ Reconnect cannot create a second billable row for the same logical tenant/provider account.
417
+
418
+ ### D14. Main deployment identity governs integration release parity
419
+
420
+ Expose a release projection containing:
421
+
422
+ ```json
423
+ {
424
+ "service": "mcp-scraper-main",
425
+ "releaseId": "...",
426
+ "integrationSchemaVersion": "...",
427
+ "providerContractVersion": "...",
428
+ "providerContractHash": "...",
429
+ "frontendReleaseId": "..."
430
+ }
431
+ ```
432
+
433
+ The scheduler exposes its consumed provider-contract hash. Production smoke fails when hashes differ or when the deployed frontend lacks Test, Refresh/Reconnect, and Disconnect controls promised by main.
434
+
435
+ ## Main MCP data model
436
+
437
+ ### `service_connections`
438
+
439
+ Authoritative public connection identity, tenant ownership, routing metadata, lifecycle, operational health, capability revision, and scheduler pointers.
440
+
441
+ ### `service_connection_credentials`
442
+
443
+ Encrypted remote-MCP credentials only. Nango credentials remain in Nango.
444
+
445
+ ### `service_connection_tool_cache`
446
+
447
+ Bounded sanitized live tool schemas keyed by connection, tool, upstream revision, and provider-contract hash. Cache entries never grant authorization by themselves.
448
+
449
+ ```sql
450
+ CREATE TABLE service_connection_tool_cache (
451
+ connection_id TEXT NOT NULL REFERENCES service_connections(id) ON DELETE CASCADE,
452
+ tool_name TEXT NOT NULL,
453
+ classification TEXT NOT NULL,
454
+ schema_json TEXT NOT NULL,
455
+ schema_hash TEXT NOT NULL,
456
+ upstream_revision TEXT,
457
+ provider_contract_hash TEXT NOT NULL,
458
+ fetched_at TEXT NOT NULL,
459
+ expires_at TEXT NOT NULL,
460
+ PRIMARY KEY(connection_id, tool_name)
461
+ );
462
+ ```
463
+
464
+ ### `service_connection_health_events`
465
+
466
+ Stores safe operational evidence without arguments or provider results.
467
+
468
+ ### `service_connection_action_audit`
469
+
470
+ Stores tenant, connection, tool, redacted argument classification, status, safe error, idempotency identity, and timestamp. It contains no provider credential or unbounded payload.
471
+
472
+ ### `service_connection_deletions`
473
+
474
+ Durable saga receipt covering upstream revocation, scheduler cleanup, local cleanup, and billing reconciliation.
475
+
476
+ ### `service_connection_migrations`
477
+
478
+ Records scheduler-to-main backfill receipts, hashes, verification, and cutover state per connection.
479
+
480
+ ## Identity migration
481
+
482
+ The scheduler currently keys connection ownership by email-like identity strings. Main ownership must use stable `users.id`.
483
+
484
+ Migration rules:
485
+
486
+ 1. Read the scheduler connection row and its legacy identity.
487
+ 2. Resolve exactly one active main user by normalized email or existing memory identity.
488
+ 3. Preserve the existing public connection UUID.
489
+ 4. Store the old identity only in `legacy_owner_identity` for audit/migration lookup.
490
+ 5. Never infer ownership when zero or multiple users match; quarantine that row for operator review.
491
+ 6. Verify Nango routing metadata without exposing it.
492
+ 7. Write a migration receipt containing safe hashes, not secrets.
493
+
494
+ No connection becomes public under a different user merely because labels or provider accounts match.
495
+
496
+ ## Scheduler data model after migration
497
+
498
+ The scheduler retains:
499
+
500
+ - `mem_scheduled_actions`;
501
+ - `mem_scheduled_action_connections` with logical public IDs and grants;
502
+ - `mem_connection_sync_checkpoints`;
503
+ - `mem_connection_sync_records`;
504
+ - typed provider tables such as Search Console performance;
505
+ - `mem_connection_sync_runs`;
506
+ - schedule and model billing receipts.
507
+
508
+ The scheduler removes or retires:
509
+
510
+ - authoritative `mem_external_connections` rows;
511
+ - Nango routing identifiers;
512
+ - remote-MCP credential rows;
513
+ - direct provider action audit;
514
+ - Nango usage authorization/settlement ownership;
515
+ - public connect/reconnect/disconnect/list/read/action/export routes.
516
+
517
+ `mem_external_connections` remains a read-only migration shadow for one rollback window, then is dropped after evidence confirms no remaining readers.
518
+
519
+ ## Public API contract
520
+
521
+ Use integration-named routes:
522
+
523
+ ```text
524
+ GET /integrations
525
+ POST /integrations/connect-session
526
+ POST /integrations/:id/reconnect-session
527
+ POST /integrations/:id/test
528
+ DELETE /integrations/:id
529
+ POST /integrations/:id/actions-enabled
530
+ POST /integrations/:id/read
531
+ POST /integrations/:id/describe
532
+ POST /integrations/:id/action
533
+ POST /integrations/:id/export
534
+ ```
535
+
536
+ Keep `/schedule-connections/*` as compatibility aliases for at least one published MCP/package window. Alias handlers call the same main-owned services; they never forward to the scheduler after cutover.
537
+
538
+ ### Public connection shape
539
+
540
+ ```ts
541
+ interface PublicServiceConnection {
542
+ connectionId: string
543
+ providerConfigKey: string
544
+ provider: string | null
545
+ label: string
546
+ transport: 'nango' | 'remote_mcp'
547
+ lifecycleStatus: ConnectionLifecycleStatus
548
+ operationalStatus: ConnectionOperationalStatus
549
+ reconnectRequired: boolean
550
+ actionsEnabled: boolean
551
+ readTools: string[]
552
+ actionTools: string[]
553
+ toolCapabilities: ToolCapability[]
554
+ lastCheckedAt: string | null
555
+ lastSuccessfulCallAt: string | null
556
+ lastFailureAt: string | null
557
+ lastFailureCode: string | null
558
+ lastFailureRetryable: boolean | null
559
+ healthEvidenceSource: string | null
560
+ schemaRevision: string | null
561
+ providerContractVersion: string
562
+ tableName: string | null
563
+ createdAt: string | null
564
+ updatedAt: string | null
565
+ }
566
+ ```
567
+
568
+ Compatibility field `status` remains temporarily and maps only to lifecycle. Documentation must not treat it as runtime health.
569
+
570
+ ## Internal scheduler-to-main API
571
+
572
+ These routes require signed scheduler service authentication and are not exposed through the root MCP:
573
+
574
+ ```text
575
+ GET /api/internal/integrations/contracts
576
+ GET /api/internal/integrations/:id/toolset
577
+ POST /api/internal/integrations/:id/read
578
+ POST /api/internal/integrations/:id/action
579
+ POST /api/internal/integrations/:id/sync-slice
580
+ POST /api/internal/integrations/:id/validate-binding
581
+ POST /api/internal/integrations/:id/report-scheduler-pointer
582
+ ```
583
+
584
+ ### Scheduled invocation shape
585
+
586
+ ```ts
587
+ interface ScheduledIntegrationInvocation {
588
+ requestId: string
589
+ scheduleId: string
590
+ occurrenceId: string
591
+ tenantIdentity: string
592
+ connectionId: string
593
+ providerConfigKey: string
594
+ tool: string
595
+ allowedTools: string[]
596
+ providerContractHash: string
597
+ args: JsonObject
598
+ issuedAt: string
599
+ expiresAt: string
600
+ nonce: string
601
+ }
602
+ ```
603
+
604
+ Main verifies every field and ignores any upstream routing value supplied by scheduler.
605
+
606
+ ## Error contract
607
+
608
+ | Code | HTTP | Lifecycle effect | Operational effect | Correct recovery |
609
+ |---|---:|---|---|---|
610
+ | `connection_not_found` | 404 | none | none | Select another connection |
611
+ | `connection_inactive` | 409 | depends on confirmed cause | unknown | Inspect lifecycle evidence |
612
+ | `provider_auth_revoked` | 401/409 | reauth required | permission limited | Reconnect |
613
+ | `missing_provider_permission` | 403 | connected | permission limited | Refresh access with required scope |
614
+ | `integration_transport_unavailable` | 503 | connected | temporarily unavailable | Retry/test; operator checks main/Nango |
615
+ | `tool_discovery_failed` | 502 | connected | provider error | Retry and inspect Nango inventory |
616
+ | `upstream_rate_limited` | 429 | connected | rate limited | Honor Retry-After |
617
+ | `provider_error` | 502 | connected | provider error | Retry or inspect provider status |
618
+ | `integration_not_configured` | 503 | connected | temporarily unavailable | Fix main deployment configuration |
619
+ | `billing_authorization_unavailable` | 503 | connected | temporarily unavailable | Retry; do not reconnect |
620
+ | `schedule_grant_invalid` | 403 | unchanged | unchanged | Scheduler refreshes binding/grant |
621
+ | `provider_contract_mismatch` | 409 | unchanged | unknown | Deploy matching contract versions |
622
+ | `actions_disabled` | 403 | connected | unchanged | Explicitly enable actions |
623
+ | `action_outcome_unknown` | 409 | connected | unchanged | Verify provider before another action |
624
+
625
+ Every error includes `requestId`, `retryable`, `sourceLayer`, and a safe message.
626
+
627
+ After cutover, external clients should receive `integration_transport_unavailable`. Accept and map legacy `connection_transport_unavailable` during the compatibility window.
628
+
629
+ ## Frontend behavior
630
+
631
+ ### Badges
632
+
633
+ | Lifecycle | Operational | Badge | Action |
634
+ |---|---|---|---|
635
+ | connected | available | Available | Test again |
636
+ | connected | unknown | Connected · Not tested | Test connection |
637
+ | connected | temporarily unavailable | Temporarily unavailable | Retry test |
638
+ | connected | rate limited | Rate limited | Retry later |
639
+ | connected | permission limited | Limited access | Refresh access |
640
+ | reauth required | any | Reconnect required | Reconnect |
641
+ | pending OAuth | unknown | Authorization pending | Continue authorization |
642
+ | deleting | any | Disconnecting | Show progress/retry state |
643
+ | cleanup pending | any | Disconnected · Cleanup pending | Contact support with receipt ID |
644
+
645
+ `Ready for sync` additionally requires recent operational availability and every required sync tool.
646
+
647
+ ### Drawer controls
648
+
649
+ Every connection transport shows:
650
+
651
+ - Test connection;
652
+ - Refresh access or Reconnect based on lifecycle;
653
+ - Disconnect;
654
+ - action switch when actions exist;
655
+ - last successful operation;
656
+ - last safe failure and request ID;
657
+ - connection public reference;
658
+ - provider-contract and schema revision under operator details.
659
+
660
+ Disconnect is never conditioned on `remote_mcp` transport.
661
+
662
+ ### Search Console outage message
663
+
664
+ For integration transport failure:
665
+
666
+ > Your Google authorization remains connected, but the main MCP could not reach the integration transport. Retrying OAuth is unlikely to help. Retry Test connection or contact support with request ID …
667
+
668
+ ## Root MCP contract
669
+
670
+ Keep existing generic tools:
671
+
672
+ - `list_service_connections`;
673
+ - `describe_service_connection_tool`;
674
+ - `read_service_connection`;
675
+ - `call_service_connection_action`;
676
+ - `export_connected_service_data`.
677
+
678
+ Add:
679
+
680
+ - `test_service_connection`.
681
+
682
+ All execute inside the main MCP integration layer. None call the scheduler unless the user explicitly creates, edits, inspects, or runs scheduled work.
683
+
684
+ Tool descriptions distinguish lifecycle from operational health. A stored row with `reconnectRequired: false` is not described as healthy without current evidence.
685
+
686
+ ## Search Console workflow contract
687
+
688
+ A comprehensive analysis begins:
689
+
690
+ 1. list main-owned service connections;
691
+ 2. select the tenant Search Console connection;
692
+ 3. inspect lifecycle and operational evidence;
693
+ 4. test the connection when health is unknown or stale;
694
+ 5. call `list-sites` directly through main;
695
+ 6. match the exact requested property by normalized site URL/domain;
696
+ 7. declare and query/export a bounded date range;
697
+ 8. validate row count, coverage, and provider warnings;
698
+ 9. enrich relevant pages and queries with scraper, SERP, PAA, Maps, or competitor data;
699
+ 10. distinguish Search Console facts from public-web inference.
700
+
701
+ Failure before valid provider rows blocks the Search Console analysis. Public data may be presented separately but cannot masquerade as a Search Console result.
702
+
703
+ ## Source movement map
704
+
705
+ ### Move or rewrite into `mcp-scraper`
706
+
707
+ | Scheduler source | Main destination/responsibility |
708
+ |---|---|
709
+ | `routes/nango-integrations.ts` | `src/api/integration-routes.ts` plus main service layer |
710
+ | `routes/resend-integrations.ts` | main remote-MCP lifecycle routes |
711
+ | `db/external-connections.ts` connection portion | Turso connection repository |
712
+ | `db/remote-mcp-connections.ts` credential portion | encrypted main credential repository |
713
+ | `lib/nango-api.ts` | main Nango API client and canonical policy input |
714
+ | `lib/nango-connection-client.ts` | MCP SDK-based main transport |
715
+ | `lib/nango-connected-mcp-catalog.ts` | main schema discovery/cache |
716
+ | `lib/nango-proxy.ts` | main Nango Proxy adapter |
717
+ | `lib/connected-data-export.ts` provider page adapters | main export execution |
718
+ | `lib/connected-usage-billing.ts` | local main ledger authorization/settlement adapter |
719
+ | connection action audit | main Turso action audit |
720
+
721
+ ### Keep and refactor in scheduler
722
+
723
+ | Scheduler source | Required change |
724
+ |---|---|
725
+ | `agents/scheduled-action-nango-tools.ts` | replace Nango handles with signed main-MCP proxy tools |
726
+ | `lib/connection-sync.ts` | call main sync-slice API; keep checkpoints/storage |
727
+ | `db/external-connections.ts` binding/sync portion | split into scheduler-owned binding and sync repositories |
728
+ | connection table/vault helpers | remain scheduler-owned storage pointers |
729
+ | scheduled action workflows/steps | remain Mastra/Inngest |
730
+ | schedule billing | retain scheduled base/model settlement; remove connected-usage ownership |
731
+
732
+ ## Secrets and configuration
733
+
734
+ ### Add to main MCP deployment
735
+
736
+ ```text
737
+ NANGO_SECRET_KEY
738
+ NANGO_MCP_URL
739
+ NANGO_API_BASE_URL
740
+ INTEGRATION_CREDENTIAL_KEY_CURRENT
741
+ INTEGRATION_CREDENTIAL_KEY_PREVIOUS
742
+ INTERNAL_INTEGRATIONS_KEY_CURRENT
743
+ INTERNAL_INTEGRATIONS_KEY_PREVIOUS
744
+ ```
745
+
746
+ ### Remove from scheduler after cutover
747
+
748
+ ```text
749
+ NANGO_SECRET_KEY
750
+ NANGO_MCP_URL
751
+ NANGO_API_BASE_URL
752
+ SCHEDULE_INTEGRATIONS_SECRET
753
+ remote-MCP credential encryption secrets
754
+ ```
755
+
756
+ The scheduler keeps only the scoped key used to sign requests to main.
757
+
758
+ Secret migration uses deployment secret-copy procedures and masked verification. Secrets never appear in logs, migration receipts, specs, or chat.
759
+
760
+ ## Reliability
761
+
762
+ ### Read retry policy
763
+
764
+ Main may retry read-only discovery, reads, and export pages for retryable transport failures:
765
+
766
+ - original attempt plus at most two retries;
767
+ - exponential backoff with jitter;
768
+ - total within the public deadline;
769
+ - honor Retry-After;
770
+ - never retry validation, permission, or confirmed auth failures.
771
+
772
+ ### Action policy
773
+
774
+ Actions require an idempotency key. Ambiguous provider outcomes return `action_outcome_unknown` and are not automatically replayed.
775
+
776
+ ### Main integration circuit breaker
777
+
778
+ Open a short provider/transport circuit after deployment-wide retryable failures cross a threshold. Fail fast without marking individual credentials for reconnect. Scheduled occurrences receive a retryable infrastructure result and retain their occurrence identity.
779
+
780
+ ### Scheduler behavior during main outage
781
+
782
+ If the main MCP is unavailable before provider execution:
783
+
784
+ - preserve the occurrence and checkpoint;
785
+ - do not double-charge scheduled base or connected usage;
786
+ - retry under the same occurrence identity;
787
+ - report an infrastructure failure, not an OAuth failure;
788
+ - do not attempt direct Nango fallback from scheduler.
789
+
790
+ There is intentionally no scheduler bypass around main ownership.
791
+
792
+ ## Security
793
+
794
+ 1. Main public authentication derives `user_id`; callers cannot submit another tenant identity.
795
+ 2. Internal scheduler calls use signed, short-lived, replay-protected assertions.
796
+ 3. Main re-resolves every connection from `connectionId + user_id`.
797
+ 4. Scheduler never supplies or receives Nango upstream routing IDs.
798
+ 5. Foreign and absent connection IDs return the same public not-found shape.
799
+ 6. Remote-MCP credentials are encrypted with versioned keys.
800
+ 7. Logs never contain credentials, tool arguments, provider payloads, Search Console rows, or prompts.
801
+ 8. Action authorization remains independent of billing.
802
+ 9. Schedule grants cannot exceed the main provider policy or account action switch.
803
+ 10. Migration scripts emit only safe hashes and counts.
804
+
805
+ ## Observability
806
+
807
+ Propagate one request ID through:
808
+
809
+ ```text
810
+ public MCP/frontend or scheduler occurrence
811
+ -> main route
812
+ -> integration executor
813
+ -> Nango/remote MCP/provider
814
+ -> usage ledger
815
+ -> health event
816
+ ```
817
+
818
+ Record:
819
+
820
+ - source surface: direct, scheduled-agent, scheduled-sync, UI-test;
821
+ - provider and operation class;
822
+ - duration and retries;
823
+ - safe result/error code;
824
+ - lifecycle and operational transitions;
825
+ - schema and provider-contract revisions;
826
+ - usage authorization/settlement receipt IDs;
827
+ - schedule/occurrence IDs for internal calls;
828
+ - migration and deletion saga states.
829
+
830
+ Alert when:
831
+
832
+ - integration failure rate affects multiple tenants;
833
+ - Nango discovery fails above threshold;
834
+ - scheduler provider-contract hash differs from main;
835
+ - migration receipts remain incomplete;
836
+ - scheduler still uses legacy Nango routes after cutover;
837
+ - deletion or cleanup sagas stall;
838
+ - recurring connection quantity differs from active main-owned Nango rows;
839
+ - frontend and main release contracts differ.
840
+
841
+ ## Migration plan
842
+
843
+ ### Phase 0 — repair current production and stop misleading users
844
+
845
+ 1. Diagnose and restore the current scheduler-hosted transport so existing customers are not left broken during migration.
846
+ 2. Deploy the current transport-neutral Disconnect frontend.
847
+ 3. Change optimistic `Ready for sync` wording to lifecycle-only `Connected` until health fields ship.
848
+ 4. Add request IDs to current proxy errors.
849
+ 5. Prove Search Console `list-sites` and bounded export using the existing connection.
850
+
851
+ Exit gate:
852
+
853
+ - current direct Search Console calls work;
854
+ - live Nango Disconnect is visible;
855
+ - no customer credential is recreated merely to fix infrastructure.
856
+
857
+ ### Phase 1 — build main-owned integration foundation
858
+
859
+ 1. Add Turso schema and repositories.
860
+ 2. Add canonical provider contracts in main.
861
+ 3. Add MCP SDK Nango and remote-MCP executors.
862
+ 4. Add local connected-usage authorization and settlement.
863
+ 5. Add health, audit, tool cache, migration, and deletion repositories.
864
+ 6. Add new `/integrations` services behind a disabled feature flag.
865
+
866
+ Feature flags:
867
+
868
+ ```text
869
+ INTEGRATION_OWNER=legacy_scheduler|shadow_main|main
870
+ INTEGRATION_MAIN_EXECUTION_ENABLED=false|true
871
+ ```
872
+
873
+ Exit gate:
874
+
875
+ - unit and contract tests pass;
876
+ - main can execute against staging connections without scheduler code;
877
+ - no production traffic is cut over.
878
+
879
+ ### Phase 2 — migrate connection metadata and credentials
880
+
881
+ 1. Create a one-time migration exporter in scheduler.
882
+ 2. Resolve each legacy identity to exactly one main user.
883
+ 3. Preserve every public connection UUID.
884
+ 4. Backfill Nango routing metadata into main.
885
+ 5. Re-encrypt remote-MCP credentials into main using the new key.
886
+ 6. Verify provider, transport, status, labels, action switch, pointers, and safe hashes.
887
+ 7. Dual-write lifecycle changes from the legacy path during the migration window.
888
+ 8. Quarantine unmatched or ambiguous owners; do not guess.
889
+
890
+ Exit gate:
891
+
892
+ - every eligible connection has a verified migration receipt;
893
+ - Nango and remote-MCP staging calls pass through main;
894
+ - no duplicate billable connection is created;
895
+ - remote-MCP plaintext is never exported.
896
+
897
+ ### Phase 3 — shadow direct reads and cut over main
898
+
899
+ Shadow only safe read-only operations:
900
+
901
+ - catalog/list comparison;
902
+ - live tool inventory comparison;
903
+ - bounded diagnostic comparison against disposable fixtures.
904
+
905
+ Do not shadow actions, OAuth sessions, reconnect, disconnect, or billed bulk export.
906
+
907
+ Then set `INTEGRATION_OWNER=main` for direct traffic. `/schedule-connections` aliases remain but execute locally in main.
908
+
909
+ Exit gate:
910
+
911
+ - scheduler deployment can be disabled in staging while direct list/test/read/describe/export works;
912
+ - direct usage settles once in main;
913
+ - health states and request traces are correct.
914
+
915
+ ### Phase 4 — convert scheduler into a main-MCP consumer
916
+
917
+ 1. Add signed internal integration client.
918
+ 2. Replace Mastra Nango tools with proxy tools.
919
+ 3. Replace deterministic sync provider calls with sync-slice requests.
920
+ 4. Backfill logical connection references in binding tables.
921
+ 5. Remove scheduler foreign-key dependence on `mem_external_connections`.
922
+ 6. Verify schedule grants against main on create/edit and execution.
923
+ 7. Retain checkpoints, records, tables, and occurrence receipts in scheduler.
924
+
925
+ Exit gate:
926
+
927
+ - scheduled agent and deterministic sync run without Nango secrets in scheduler;
928
+ - same provider-contract hash is used;
929
+ - schedule retries do not double bill;
930
+ - existing bindings still target the preserved public IDs.
931
+
932
+ ### Phase 5 — delete legacy ownership from scheduler
933
+
934
+ 1. Remove public integration control routes from scheduler.
935
+ 2. Remove Nango and remote-MCP secrets from scheduler deployment.
936
+ 3. Stop dual writes.
937
+ 4. Freeze `mem_external_connections` as read-only for one rollback window.
938
+ 5. Prove zero legacy readers through telemetry.
939
+ 6. Delete migrated remote-MCP credential ciphertext from scheduler.
940
+ 7. Drop legacy connection tables only after backup retention and migration sign-off.
941
+ 8. Remove `NANGO_CONTROL_URL` and `SCHEDULE_INTEGRATIONS_SECRET` from main.
942
+
943
+ Exit gate:
944
+
945
+ - scheduler is independently deployable as a Mastra scheduling system;
946
+ - direct integrations are entirely main-owned;
947
+ - no legacy route traffic or secret remains.
948
+
949
+ ## Repository task breakdown
950
+
951
+ ### `mcp-scraper`
952
+
953
+ - MI-01: Add main integration schema and migration registry entries.
954
+ - MI-02: Add stable user-owned connection repository.
955
+ - MI-03: Add encrypted remote-MCP credential repository and key rotation.
956
+ - MI-04: Move canonical provider contracts into main and generate scheduler projection.
957
+ - MI-05: Implement MCP SDK Nango client and live tool discovery.
958
+ - MI-06: Implement remote-MCP client and OAuth/API-key lifecycle.
959
+ - MI-07: Implement main read/action/describe/export executor.
960
+ - MI-08: Integrate connected usage with the local Credit ledger.
961
+ - MI-09: Implement lifecycle and operational health state.
962
+ - MI-10: Implement action audit, health events, and request traces.
963
+ - MI-11: Implement Test connection and Search Console safe probe.
964
+ - MI-12: Implement idempotent disconnect saga and cleanup retries.
965
+ - MI-13: Implement reconnect identity preservation and duplicate prevention.
966
+ - MI-14: Add signed scheduler invocation verification and replay protection.
967
+ - MI-15: Add `/integrations` routes and compatibility aliases.
968
+ - MI-16: Update root MCP schemas, instructions, and `test_service_connection`.
969
+ - MI-17: Update frontend states and universal lifecycle controls.
970
+ - MI-18: Add release/provider-contract parity metadata.
971
+ - MI-19: Add migration import and verification tooling.
972
+ - MI-20: Update docs, MCPB, generated manifest, SDK types, and legal surfaces.
973
+
974
+ ### `mcp-scraper-scheduler`
975
+
976
+ - MS-01: Split scheduler-owned bindings/sync state from legacy connection repository.
977
+ - MS-02: Add migration exporter with safe hashes and no plaintext output.
978
+ - MS-03: Add signed main-MCP integration client.
979
+ - MS-04: Replace direct Nango Mastra tools with proxy tools.
980
+ - MS-05: Replace deterministic provider execution with sync-slice calls.
981
+ - MS-06: Migrate bindings to logical public IDs and remove connection foreign key.
982
+ - MS-07: Validate provider-contract hash and exact grants.
983
+ - MS-08: Add idempotent connection purge endpoint for disconnect saga.
984
+ - MS-09: Remove connected-usage settlement ownership.
985
+ - MS-10: Remove public integration routes, credentials, and legacy tables after proof.
986
+
987
+ ## Testing
988
+
989
+ ### Main unit tests
990
+
991
+ - user ownership and foreign-ID nondisclosure;
992
+ - connection lifecycle/operational transitions;
993
+ - provider contract generation/hash;
994
+ - MCP SDK schema sanitization;
995
+ - Nango and remote-MCP transport selection;
996
+ - encrypted credential round-trip and key rotation;
997
+ - read retry eligibility and deadline budget;
998
+ - action idempotency and ambiguous outcomes;
999
+ - health TTL expiry;
1000
+ - Search Console list-sites probe;
1001
+ - local connected-usage debit idempotency;
1002
+ - signed scheduler request verification, expiry, and replay rejection;
1003
+ - disconnect saga states and retry;
1004
+ - reconnect preserves public ID and quantity.
1005
+
1006
+ ### Scheduler unit tests
1007
+
1008
+ - proxy tool schema and main endpoint execution;
1009
+ - exact grant enforcement before dispatch;
1010
+ - provider-contract hash mismatch fails closed;
1011
+ - occurrence identity survives retries;
1012
+ - sync-slice checkpoint persistence;
1013
+ - purge endpoint removes binding/sync state idempotently;
1014
+ - no code path opens a Nango client after cutover.
1015
+
1016
+ ### Cross-repo contract tests
1017
+
1018
+ - scheduler projection matches main provider-contract hash;
1019
+ - signed invocation schemas match;
1020
+ - normalized result and error schemas match;
1021
+ - old and new public route aliases are equivalent;
1022
+ - main release contract matches frontend bundle;
1023
+ - root MCP tool manifest contains health fields and Test connection;
1024
+ - scheduler package/deployment contains no Nango secret requirement in main-owner mode.
1025
+
1026
+ ### Migration tests
1027
+
1028
+ - exact email-to-user match;
1029
+ - missing user quarantine;
1030
+ - ambiguous user quarantine;
1031
+ - public UUID preservation;
1032
+ - Nango routing metadata backfill;
1033
+ - remote credential re-encryption without plaintext logs;
1034
+ - action switch and table/vault pointer preservation;
1035
+ - dual-write conflict detection;
1036
+ - rollback reads legacy shadow without overwriting main.
1037
+
1038
+ ### End-to-end staging tests
1039
+
1040
+ 1. Disable scheduler and prove direct Search Console list/test/read/export through main.
1041
+ 2. Disable main integration execution and prove scheduler fails safely without Nango fallback.
1042
+ 3. Run a scheduled Mastra agent with proxied provider tools.
1043
+ 4. Run deterministic Search Console sync through sync-slice.
1044
+ 5. Revoke a Google grant and verify only main marks reauth required.
1045
+ 6. Reconnect and verify the same public connection and bindings.
1046
+ 7. Disconnect a disposable fixture and verify upstream, main, scheduler, and billing cleanup exactly once.
1047
+ 8. Rotate internal signing and remote credential keys.
1048
+ 9. Verify request trace and usage receipt continuity.
1049
+
1050
+ ### Production proof
1051
+
1052
+ For the existing Search Console tenant, record sanitized evidence only:
1053
+
1054
+ - main and scheduler release IDs;
1055
+ - matching provider-contract hashes;
1056
+ - main-owned public connection ID;
1057
+ - lifecycle and operational states;
1058
+ - successful `list-sites` count and exact matched property URL;
1059
+ - bounded export count/date range/warnings;
1060
+ - UI screenshot with Test, Refresh/Reconnect, and Disconnect;
1061
+ - direct request trace containing no scheduler hop;
1062
+ - scheduled request trace showing scheduler-to-main hop;
1063
+ - one connected-usage receipt per provider operation;
1064
+ - unchanged recurring connection quantity during migration.
1065
+
1066
+ ## Acceptance criteria
1067
+
1068
+ The change is complete only when:
1069
+
1070
+ 1. Main Turso is authoritative for every connection record.
1071
+ 2. Main owns Nango and remote-MCP execution.
1072
+ 3. Main owns encrypted remote-MCP credential storage.
1073
+ 4. Scheduler contains no Nango or provider credential.
1074
+ 5. Scheduler downtime does not affect direct integrations.
1075
+ 6. Direct integration requests contain no scheduler network hop.
1076
+ 7. Scheduler uses signed main-MCP proxy tools for scheduled provider work.
1077
+ 8. Mastra and Inngest remain scheduler-owned.
1078
+ 9. Scheduler checkpoints, synchronized records, and run receipts remain intact.
1079
+ 10. Existing public connection IDs and schedule bindings are preserved.
1080
+ 11. No healthy customer connection is forced to reauthorize during migration.
1081
+ 12. Lifecycle and operational health are distinct everywhere.
1082
+ 13. A transport 503 never becomes Reconnect required.
1083
+ 14. Search Console Test connection runs directly through main.
1084
+ 15. Geny's Flowers property selection uses an exact accessible property URL.
1085
+ 16. Refresh/Reconnect and Disconnect are visible for Nango connections.
1086
+ 17. Disconnect revokes upstream access and cleans main, scheduler, and billing state exactly once.
1087
+ 18. Reconnect preserves logical identity and billable quantity.
1088
+ 19. Main is the sole provider-policy source.
1089
+ 20. Scheduler fails closed on a provider-contract hash mismatch.
1090
+ 21. Main is the sole connected-usage debit authority and executor.
1091
+ 22. Scheduled base/model billing remains correctly separated.
1092
+ 23. Production frontend and backend releases are in parity.
1093
+ 24. Legacy scheduler integration routes show zero traffic before removal.
1094
+ 25. All unit, contract, migration, staging, and production proof gates pass.
1095
+
1096
+ ## Rollback
1097
+
1098
+ Rollback is phase-specific:
1099
+
1100
+ ### Before direct cutover
1101
+
1102
+ Disable main execution and retain legacy scheduler proxying. Keep additive main tables and migration receipts.
1103
+
1104
+ ### After direct cutover but before scheduler conversion
1105
+
1106
+ Set `INTEGRATION_OWNER=legacy_scheduler` for read-only direct operations. Do not fail over OAuth sessions, reconnect, actions, or disconnect automatically because duplicate side effects are possible.
1107
+
1108
+ ### After scheduler conversion
1109
+
1110
+ Scheduler may temporarily use the read-only legacy shadow only if the legacy Nango secrets have not yet been removed and an explicit rollback is approved. It must never dynamically bypass main during an outage.
1111
+
1112
+ ### After secret removal
1113
+
1114
+ Rollback means redeploying a verified previous release with intentionally restored secrets, not silently reactivating stale scheduler code. Completed reconnects and disconnects are never replayed.
1115
+
1116
+ Health truth, request IDs, and the corrected frontend must not be rolled back even when execution ownership is temporarily reverted.
1117
+
1118
+ ## Risks and mitigations
1119
+
1120
+ | Risk | Mitigation |
1121
+ |---|---|
1122
+ | Turso becomes authoritative for more integration state | Additive migrations, explicit repository tests, stable user IDs, migration receipts |
1123
+ | Remote credentials move databases | Envelope encryption, key rotation, no plaintext exporter, verified deletion from scheduler |
1124
+ | Scheduler loses direct provider access | Signed proxy tools, bounded deadlines, retryable occurrence semantics, no hidden fallback |
1125
+ | Cross-system grant spoofing | HMAC request signing, replay protection, main ownership check, exact grant and contract hash validation |
1126
+ | Scheduled latency increases | Same-region calls, connection reuse, schema caching, sync-slice batching |
1127
+ | Duplicate billing during cutover | One owner flag, occurrence-scoped idempotency, shadowing excludes billed calls |
1128
+ | Public IDs change | Preserve scheduler UUIDs as main primary keys |
1129
+ | Binding FK removal weakens local integrity | Main validation on create/edit/execute plus scheduler cleanup saga |
1130
+ | Source policy drifts | Main canonical contract and generated scheduler projection |
1131
+ | Main outage affects schedules | Correct dependency: scheduler retries; it does not take over integration ownership |
1132
+ | Frontend remains stale | Release identity and live bundle contract smoke |
1133
+
1134
+ ## Evidence anchors
1135
+
1136
+ ### Main MCP
1137
+
1138
+ - `src/api/db.ts`: Turso/libSQL user, billing, workflow, and migration authority.
1139
+ - `src/api/nango-control.ts`: current main-to-scheduler connection proxy and legacy error mapping.
1140
+ - `src/api/server.ts`: public connection, lifecycle, read/action/export, binding, and billing routes.
1141
+ - `src/api/connected-data-export.ts`: aggregate export orchestration already in main.
1142
+ - `src/api/connected-account-billing.ts`: recurring connection quantity contract.
1143
+ - `src/api/connected-usage-billing.ts`: authoritative main connected-usage boundary.
1144
+ - `src/mcp/http-mcp-tool-executor.ts`: root MCP bridge.
1145
+ - `src/mcp/mcp-tool-schemas.ts`: public generic integration schemas.
1146
+ - `public/app.jsx` and `public/app.js`: main integration frontend and current local transport-neutral Disconnect.
1147
+
1148
+ ### Scheduler
1149
+
1150
+ - `src/mastra/routes/nango-integrations.ts`: accidental Nango integration API owner.
1151
+ - `src/mastra/routes/resend-integrations.ts`: accidental remote-MCP lifecycle owner.
1152
+ - `src/mastra/db/external-connections.ts`: mixed integration and scheduling persistence.
1153
+ - `src/mastra/db/remote-mcp-connections.ts`: scheduler-held remote credentials.
1154
+ - `src/mastra/lib/nango-api.ts`: provider catalog, policy, Nango API, and routing.
1155
+ - `src/mastra/lib/nango-connection-client.ts`: Mastra MCP connection client to replace in main with standard MCP SDK.
1156
+ - `src/mastra/agents/scheduled-action-nango-tools.ts`: direct provider tools to convert into main-MCP proxies.
1157
+ - `src/mastra/lib/connection-sync.ts`: deterministic sync orchestration that remains scheduler-owned.
1158
+ - `src/mastra/lib/connected-usage-billing.ts`: scheduler-to-main billing client to retire for provider execution.
1159
+
1160
+ ## Final architectural statement
1161
+
1162
+ Integrations are part of MCP Scraper's main product surface and security boundary.
1163
+
1164
+ The main MCP owns the connection, credential route, provider policy, provider call, health truth, lifecycle mutation, audit, export, and connected-usage charge. The Mastra scheduler owns when scheduled work runs, how it is orchestrated, and where scheduled state is stored. It invokes integrations through main like any other authorized internal consumer; it never becomes the integration backend again.