mcp-scraper 0.15.0 → 0.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -5
- package/dist/bin/api-server.cjs +278 -20
- package/dist/bin/api-server.cjs.map +1 -1
- package/dist/bin/api-server.js +1 -1
- package/dist/bin/mcp-scraper-cli.cjs +1 -1
- package/dist/bin/mcp-scraper-cli.cjs.map +1 -1
- package/dist/bin/mcp-scraper-cli.js +1 -1
- package/dist/bin/mcp-scraper-install.cjs +1 -1
- package/dist/bin/mcp-scraper-install.cjs.map +1 -1
- package/dist/bin/mcp-scraper-install.js +1 -1
- package/dist/bin/mcp-stdio-server.cjs +42 -10
- package/dist/bin/mcp-stdio-server.cjs.map +1 -1
- package/dist/bin/mcp-stdio-server.js +2 -2
- package/dist/chunk-HQYIP5X3.js +7 -0
- package/dist/chunk-HQYIP5X3.js.map +1 -0
- package/dist/{chunk-ETJBTYZX.js → chunk-YIV4IKFG.js} +43 -11
- package/dist/chunk-YIV4IKFG.js.map +1 -0
- package/dist/{server-DDKXWYTO.js → server-QEXOVJPR.js} +238 -12
- package/dist/server-QEXOVJPR.js.map +1 -0
- package/docs/mcp-tool-manifest.generated.json +162 -6
- package/docs/specs/google-drive-bulk-access-and-mcp-schema-passthrough-spec.md +1680 -0
- package/package.json +1 -1
- package/dist/chunk-ETJBTYZX.js.map +0 -1
- package/dist/chunk-TK2S2M7G.js +0 -7
- package/dist/chunk-TK2S2M7G.js.map +0 -1
- package/dist/server-DDKXWYTO.js.map +0 -1
|
@@ -0,0 +1,1680 @@
|
|
|
1
|
+
# Google Drive Bulk Access and MCP Schema Passthrough
|
|
2
|
+
|
|
3
|
+
- Status: Phase 1 interactive tools live; deterministic bulk, native connection MCP, broader writes, and Google OAuth review remain gated
|
|
4
|
+
- Spec date: 2026-07-12
|
|
5
|
+
- Primary repos: `mcp-scraper`, `mcp-scraper-scheduler`, `mcpscraper-web`, `mcpscraper-sdk`
|
|
6
|
+
- Primary provider key: `google-drive`
|
|
7
|
+
|
|
8
|
+
## Phase 1 production receipt — 2026-07-12
|
|
9
|
+
|
|
10
|
+
- Nango production has the exact source-controlled 12-function Drive inventory: ten bounded reads plus the additive `create-drive-folder` and `copy-drive-file` actions.
|
|
11
|
+
- The scheduler exposes the live-policy intersection as 10 reads and 2 actions, injects exact per-schedule grants into transient Mastra toolsets, and audits classified mutations without storing argument values.
|
|
12
|
+
- The current tenant-owned Drive connection is healthy. Its explicitly authorized action switch is on; newly connected accounts still default to actions off.
|
|
13
|
+
- The fixed public MCP remains a 156-tool provider-neutral surface. `describe_service_connection_tool` returned live 64-character schema hashes for both a Drive read and action; `get-about` and a one-item `list-drive-files` page passed through production.
|
|
14
|
+
- This receipt covers interactive Phase 1 only. Deterministic `connection_sync`, background Proxy/Blob file transfer, native connection-scoped MCP `tools/list`, `upload-document`, broader mutations, and a reversible production write fixture are not shipped.
|
|
15
|
+
|
|
16
|
+
## Executive decision
|
|
17
|
+
|
|
18
|
+
Google Drive becomes the first complete connected-data provider with two deliberately separate execution paths:
|
|
19
|
+
|
|
20
|
+
1. **Interactive tools:** tenant-scoped, schema-discoverable tools for small live reads and separately gated writes.
|
|
21
|
+
2. **Deterministic bulk:** a checkpointed background export/sync service that inventories files, follows Drive change tokens, streams downloadable content through Nango Proxy into private object storage, and returns manifests rather than file bytes in MCP JSON.
|
|
22
|
+
|
|
23
|
+
Nango remains the production OAuth and credential boundary for the REST/bulk path. The optional Google remote MCP has a separate OAuth credential boundary. The Google Drive REST API, reached through Nango Proxy, is the authoritative path for complete inventory, incremental reconciliation, and large downloads. Google's official remote Drive MCP is useful as a feature-flagged interactive provider and protocol-conformance target, but it is Developer Preview, exposes only eight tools, filters files for generative-AI eligibility, and returns downloads as base64. It must not be the bulk-transfer foundation.
|
|
24
|
+
|
|
25
|
+
MCP does not define a `tools/describe` RPC. The proper protocol surface is `tools/list`, where each Tool already carries its input schema, optional output schema, description, and annotations. MCP Scraper will therefore provide:
|
|
26
|
+
|
|
27
|
+
- a **protocol-native connection MCP endpoint** whose `tools/list` projects the selected connection's callable provider tools; and
|
|
28
|
+
- a backward-compatible ordinary tool, `describe_service_connection_tool`, for agents using the fixed root MCP and its generic read/action bridges.
|
|
29
|
+
|
|
30
|
+
The compatibility tool is useful, but it must never be described as a new MCP protocol method.
|
|
31
|
+
|
|
32
|
+
## Canonical Build Spec
|
|
33
|
+
|
|
34
|
+
- **Request:** Make Google Drive operational first, including large/bulk downloads, scheduled ingestion, full live schema discovery, and safe read/write expansion.
|
|
35
|
+
- **Lane:** MCP Build using Mastra, with Mastra Build for deterministic scheduled execution.
|
|
36
|
+
- **Capability:** Google Drive Bulk Access + Connection-Scoped MCP Tool Projection.
|
|
37
|
+
- **Audience:** MCP Scraper users connecting personal or workspace Google Drives; MCP clients; Mastra scheduled agents; operators reviewing production readiness.
|
|
38
|
+
- **User-Facing Outcome:** A user connects their own Drive, asks for a bounded corpus such as “give me the last 7 days of Drive files,” receives a private manifest and downloadable artifacts without repeated tool calls, and can grant exact Drive tools to a scheduled action.
|
|
39
|
+
- **Target Runtime:** Vercel-hosted MCP Scraper API and HTTP MCP; Vercel-hosted Mastra scheduler with Inngest; Nango Cloud OAuth/MCP/Proxy; Postgres scheduler state; private Vercel Blob delivery.
|
|
40
|
+
- **Inputs:** Tenant identity, tenant-owned connection ID, access profile, Drive selection, time range or Drive query, content policy, export formats, byte/file/call limits, exact scheduled tool grants.
|
|
41
|
+
- **Source Materials:** Current main/scheduler code, live sanitized production inventory, MCP stable specification 2025-11-25, Google Drive API and remote MCP documentation, Nango Drive/Proxy/limits documentation, Vercel Blob documentation.
|
|
42
|
+
- **Missing Context:** Final Google Cloud OAuth verification status; whether the existing Cloud project is on Google's May 2026 quota model; proven Nango Proxy behavior for Range and redirected Drive long-running download URLs; proven cross-invocation Blob multipart resume; final customer retention/entitlement policy.
|
|
43
|
+
- **Questions Asked:** Is a custom describe method proper MCP? Can Nango Actions carry large files? Can Google's remote MCP replace Drive REST for complete exports?
|
|
44
|
+
- **Answers Received:** No, native discovery is `tools/list`; no, Nango Actions cap output at 2 MB; no, Google's remote MCP is interactive/preview and not a complete or resumable inventory path.
|
|
45
|
+
- **Evidence Plan:** Source audit, sanitized live inventory, official documentation, isolated unit/contract tests, staging OAuth/reconnect, provider tool discovery, large-file streaming test, shared-drive pagination test, scheduled occurrence receipt, production smoke.
|
|
46
|
+
- **Evidence Acquired:** The original baseline had an auth-failed connection and zero live Drive functions. Phase 1 now has a healthy tenant connection, 12 deployed source-controlled functions, a 10-read/2-action live intersection, rich compatibility schemas, two successful bounded production reads, audited scheduled-action injection, and an explicitly enabled current-account action gate. Drive export/sync state remains absent by design.
|
|
47
|
+
- **Agents:** Scheduled Action Agent for interactive schedules; deterministic connection-sync worker for automatic ingestion; no LLM in bulk traversal or byte transfer.
|
|
48
|
+
- **Skills:** `mastra-agentic-development` for capability compilation and release receipts; full MCP release workflow for final cross-repo deployment.
|
|
49
|
+
- **Workflows:** Connect/reconnect; live tool discovery; bounded live read; gated action; initial Drive snapshot; change-token reconciliation; content transfer; scheduled Drive sync; artifact renewal/cleanup; schema-change invalidation.
|
|
50
|
+
- **Tools:** Existing generic connection tools, extended generic export-job tools, provider-native connection-scoped Drive tools, internal REST/Proxy adapters.
|
|
51
|
+
- **MCP:** Root fixed MCP remains provider-neutral; connection MCP exposes dynamic tools through `tools/list`; compatibility describe returns a sanitized Tool projection; v1 remains stateless and does not advertise `tools.listChanged`.
|
|
52
|
+
- **Data Shape:** Connection capability snapshot, export job, checkpoint, file manifest record, private artifact record, schema snapshot, schedule run receipt.
|
|
53
|
+
- **Signals:** OAuth connected/reconnected/failed; tool inventory changed; export queued/progress/partial/complete/failed/cancelled; schedule occurrence claimed/completed; artifact expiring/expired.
|
|
54
|
+
- **Background Tasks:** Initial inventory, changes reconciliation, per-file transfer, Blob multipart upload, manifest finalization, artifact cleanup, schema cache refresh.
|
|
55
|
+
- **Guardrails:** Tenant ownership, live-policy intersection, exact schedule grant, action switch, destructive-action denylist, restricted-scope disclosure, remote-MCP eligibility awareness, explicit REST `canDownload`/restriction/encryption/error skip policy, prompt-injection treatment, SSRF host allowlist, no credentials or raw provider download URLs in results.
|
|
56
|
+
- **Observability:** Per-job counts/bytes/calls/retries, per-file typed status, tool/schema revision, schedule quota receipt, structured safe error codes, Nango/Drive request traces without content or tokens.
|
|
57
|
+
- **Enforcement:** Fail closed when live inventory and policy differ; reauthorize after scope changes; never return large bytes through MCP or Nango Actions; one scheduled occurrence consumes one scheduled-event unit; release fails on surface/inventory drift.
|
|
58
|
+
- **Persistence:** Nango owns credentials; scheduler Postgres owns routes, checkpoints, records, export jobs, and run receipts; private Blob owns content and manifests; database stores artifact references, never binary bodies.
|
|
59
|
+
- **Deployment:** Source-controlled Nango functions, scheduler, main API/MCP, frontend, SDK/package, generated manifest/MCPB, Vercel production, npm production, live proof.
|
|
60
|
+
- **Done Contract:** A reauthorized Drive connection exposes the exact live approved toolset; a last-seven-days export paginates My Drive and shared drives without model loops; large content streams to owner-scoped private artifacts; a scheduled `connection_sync` resumes from Drive change tokens and bills once; both custom describe and native `tools/list` return valid sanitized schemas; all surfaces and production receipts agree.
|
|
61
|
+
- **Phase Gates:** Detailed in “Release sequence and phase gates.”
|
|
62
|
+
- **Receipt Requirements:** Commit SHAs, Nango deploy inventory, sanitized connection/tool proof, test output, Vercel deployment IDs, npm versions, MCP tool count/manifest parity, sample export/job/Blob/schedule receipts.
|
|
63
|
+
- **Remaining Gaps:** Raw paginated Nango MCP catalog capture is proven for Phase 1. Nango Range/LRO streaming and cross-invocation multipart resume remain unproven; Google OAuth verification/CASA is an external gate; official Drive MCP remains preview; final storage/egress entitlements require a product decision.
|
|
64
|
+
|
|
65
|
+
## Product contract in plain language
|
|
66
|
+
|
|
67
|
+
The following user experiences must work:
|
|
68
|
+
|
|
69
|
+
### Connect Drive
|
|
70
|
+
|
|
71
|
+
The user opens Integrations, clicks Google Drive, chooses an access profile, approves Google OAuth, and returns to a connected card that reports what is actually callable. Connections are isolated by tenant and account. No other user shares the credential or its Drive data.
|
|
72
|
+
|
|
73
|
+
The first production profile is **Full Drive read** because complete inventory and bulk download require `drive.readonly`. A Picker-limited profile can be offered separately, but it must say that it does not cover the whole Drive.
|
|
74
|
+
|
|
75
|
+
### Ask for a bounded corpus
|
|
76
|
+
|
|
77
|
+
Example: “Give me everything modified in Google Drive during the last seven days, including file contents where possible.”
|
|
78
|
+
|
|
79
|
+
One call to `export_connected_service_data` validates the request and either:
|
|
80
|
+
|
|
81
|
+
- completes a small metadata-only export inline; or
|
|
82
|
+
- creates an idempotent background export job and returns `jobId`, progress/status, preview metadata, and eventually a private manifest artifact.
|
|
83
|
+
|
|
84
|
+
The model never loops once per file. The server owns enumeration, pagination, retries, change reconciliation, content eligibility, transfers, and manifests.
|
|
85
|
+
|
|
86
|
+
### Download a folder or a large corpus
|
|
87
|
+
|
|
88
|
+
The server streams each eligible file to a private Blob artifact. The MCP result contains metadata and artifact handles, not base64. The manifest records every success, skip, failure, and continuation state. A 5 GB file is a transfer job, not a tool response.
|
|
89
|
+
|
|
90
|
+
### Schedule ingestion
|
|
91
|
+
|
|
92
|
+
The user can create a `connection_sync` schedule and bind the Drive connection. At the due time, the scheduler claims one occurrence, performs a bounded deterministic slice, saves its checkpoint, and emits a receipt. Internal pages and file downloads do not each count as scheduled events. Retrying the same leased occurrence does not double bill.
|
|
93
|
+
|
|
94
|
+
### Use provider-native schemas
|
|
95
|
+
|
|
96
|
+
An MCP client pointed at the connection-scoped endpoint receives the Drive tools directly in `tools/list`. A client using MCP Scraper's fixed root endpoint can call `describe_service_connection_tool` before using the generic read/action bridge.
|
|
97
|
+
|
|
98
|
+
## Pre-implementation baseline
|
|
99
|
+
|
|
100
|
+
This table records the state observed before Phase 1 implementation. The production receipt above is the current source of truth.
|
|
101
|
+
|
|
102
|
+
| Layer | Current Google Drive state | Required state |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| OAuth/Nango catalog | One integration; one connection with auth error / `reauth_required` | Production credentials, explicit scope profile, active reauthorized connection |
|
|
105
|
+
| Live Nango functions | 0 reads, 0 actions | Source-controlled deployed functions; live catalog matches policy |
|
|
106
|
+
| Scheduler static policy | Five legacy reads, zero actions | Policy generated/reviewed against the deployed function manifest |
|
|
107
|
+
| Mastra agent schedule | Generic injection works but Drive injects nothing | Exact connection-scoped Drive tools bind into eligible runs |
|
|
108
|
+
| Deterministic sync | Drive unsupported | Initial snapshot + Drive Changes adapter with checkpoints |
|
|
109
|
+
| Bulk export | Gmail/Calendar/Zoom/Resend only | Drive datasets and background job path |
|
|
110
|
+
| Blob delivery | Private 7-day JSONL artifacts already work | Reuse for manifest; add per-file private streaming/multipart artifacts |
|
|
111
|
+
| Root MCP describe | Resend only; Nango returns 501 | Nango and remote-MCP transports supported |
|
|
112
|
+
| Native schema discovery | No connection-scoped dynamic MCP | Connection endpoint implements native `tools/list`/`tools/call` |
|
|
113
|
+
| Frontend | Connected card can say “scheduler tools pending” | Scope-aware connect/reconnect, real readiness, export/schedule controls |
|
|
114
|
+
| SDK/CLI | No connected-service SDK contract | Typed connection/export/schema/job client in public SDK |
|
|
115
|
+
|
|
116
|
+
Existing foundations to reuse:
|
|
117
|
+
|
|
118
|
+
- exact tenant-owned connection records without storing credentials;
|
|
119
|
+
- live Nango tool discovery and per-schedule grants;
|
|
120
|
+
- deterministic `connection_sync` run claiming, quotas, leases, checkpoints, records, and receipts;
|
|
121
|
+
- private Blob artifacts with owner checks, seven-day retention, 15-minute signed URLs, renewal, and cleanup;
|
|
122
|
+
- generic MCP list/read/action/export bridges.
|
|
123
|
+
|
|
124
|
+
## Architectural decisions
|
|
125
|
+
|
|
126
|
+
### D1. Nango + Drive REST is the production bulk path
|
|
127
|
+
|
|
128
|
+
Nango handles OAuth and token refresh. Inventory and transfer calls use the Nango Proxy so MCP Scraper never handles long-lived Google credentials.
|
|
129
|
+
|
|
130
|
+
Nango Actions must not carry file bytes. Nango Cloud limits action output to 2 MB and explicitly directs large payloads to Proxy. Action/functions may return metadata or create a job; they cannot be the content channel.
|
|
131
|
+
|
|
132
|
+
### D2. Google's remote Drive MCP is optional interactive infrastructure
|
|
133
|
+
|
|
134
|
+
Google's Developer Preview endpoint is `https://drivemcp.googleapis.com/mcp/v1`. It currently documents eight tools:
|
|
135
|
+
|
|
136
|
+
- reads: `search_files`, `list_recent_files`, `get_file_metadata`, `get_file_permissions`, `read_file_content`, `download_file_content`;
|
|
137
|
+
- writes: `create_file`, `copy_file`.
|
|
138
|
+
|
|
139
|
+
It is the first external conformance target for generic schema passthrough because it exposes live MCP Tool schemas. It is not the authoritative bulk path because:
|
|
140
|
+
|
|
141
|
+
- `download_file_content` returns base64 in MCP JSON;
|
|
142
|
+
- there are no change tokens, full corpora traversal, Range checkpoints, or bulk job primitives;
|
|
143
|
+
- Developer Preview eligibility filters can omit IRM/CAA/CSE/malware/trash content that the user can otherwise see;
|
|
144
|
+
- the exposed tool surface can change before general availability.
|
|
145
|
+
|
|
146
|
+
V1 raw discovery may verify all eight upstream schemas, but the public connection policy hard-blocks `download_file_content` because its unbounded base64 result violates the large-payload contract. `read_file_content` has a gateway response-byte cap and remains for bounded interactive use. Users requesting bytes are routed to `export_connected_service_data`. The two writes remain hidden until action scope/switch/policy permit them.
|
|
147
|
+
|
|
148
|
+
Ship this adapter behind `GOOGLE_DRIVE_REMOTE_MCP_ENABLED=false` by default. Enabling it does not remove or replace the Nango Drive connection.
|
|
149
|
+
|
|
150
|
+
The default product connection remains the Nango `google-drive` row. A Google remote-MCP credential, when explicitly enabled, is a separate `remote_mcp` connection with provider key `google-drive-remote-mcp-preview` and must be labeled as interactive/preview with bulk and scheduled-sync support disabled. Do not silently create two Drive credentials from one Connect click or imply that the preview connection is the bulk connection.
|
|
151
|
+
|
|
152
|
+
Google's current preview setup asks the OAuth client for both `drive.readonly` and `drive.file`. Treat that consent contract independently from the Nango profiles below, and still hide `create_file`/`copy_file` until MCP Scraper's action switch and policy permit them.
|
|
153
|
+
|
|
154
|
+
### D3. One connection has one credential owner
|
|
155
|
+
|
|
156
|
+
Every call resolves:
|
|
157
|
+
|
|
158
|
+
```text
|
|
159
|
+
authenticated MCP Scraper identity
|
|
160
|
+
-> tenant-owned external connection row
|
|
161
|
+
-> provider config + connection id
|
|
162
|
+
-> Nango credential or remote-MCP OAuth storage
|
|
163
|
+
-> live provider tool/API call
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
A connection ID is routing input, never authorization. Missing and foreign connections return the same public 404 shape.
|
|
167
|
+
|
|
168
|
+
### D4. Large data is a job and manifest, not model context
|
|
169
|
+
|
|
170
|
+
The model receives only high-signal progress, previews, counts, warnings, and artifact handles. File bodies go from Drive through Nango Proxy to private Blob. The manifest is JSONL and references per-file artifact IDs.
|
|
171
|
+
|
|
172
|
+
Do not build a giant in-memory ZIP in Vercel v1. A manifest plus individual artifacts is resumable, deduplicable, and failure-transparent. A bundle job can be added later.
|
|
173
|
+
|
|
174
|
+
### D5. Native MCP schemas come from `tools/list`
|
|
175
|
+
|
|
176
|
+
There is no custom JSON-RPC `tools/describe`. The gateway asks the upstream connection for `tools/list`, intersects it with MCP Scraper policy, sanitizes it, and either:
|
|
177
|
+
|
|
178
|
+
- returns the Tool objects from the connection MCP's own `tools/list`; or
|
|
179
|
+
- returns one selected Tool through the ordinary compatibility tool.
|
|
180
|
+
|
|
181
|
+
Provider annotations are display/planning hints, not authorization. Classification and callability come from MCP Scraper policy and account state.
|
|
182
|
+
|
|
183
|
+
### D6. Exposure is an intersection, not a union
|
|
184
|
+
|
|
185
|
+
For every surface:
|
|
186
|
+
|
|
187
|
+
```text
|
|
188
|
+
callable tools
|
|
189
|
+
= live upstream tools
|
|
190
|
+
intersect source-controlled read/action classification
|
|
191
|
+
intersect active tenant-owned connection
|
|
192
|
+
minus administrative denylist
|
|
193
|
+
intersect per-account action switch for mutations
|
|
194
|
+
intersect per-schedule grant when scheduled
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
Catalog, UI, root MCP, connection MCP, scheduler binding, and execution validation must all consume the same resolved capability snapshot.
|
|
198
|
+
|
|
199
|
+
## System boundaries
|
|
200
|
+
|
|
201
|
+
| Component | Owns | Must not own |
|
|
202
|
+
|---|---|---|
|
|
203
|
+
| Nango | OAuth credentials, token refresh, authenticated Proxy, deployed small functions | Bulk file bodies, product billing, tenant job manifests |
|
|
204
|
+
| Mastra scheduler | Scheduled occurrence, export job state/execution, deterministic traversal, checkpoints, exact tool grants, private Blob writes, run receipts | Provider credentials, public/signed artifact authorization |
|
|
205
|
+
| MCP Scraper main API | User auth, generic MCP/API contract, job create/status/cancel facade, owner-checked signed artifact delivery | Direct long-lived Google credentials, long-running transfer execution |
|
|
206
|
+
| Private Blob | Per-file content and JSONL manifests | Authorization decisions, provider tokens |
|
|
207
|
+
| Root MCP | Stable generic connection/job tools | Hundreds of always-on provider schemas |
|
|
208
|
+
| Connection MCP | Dynamic callable provider tool projection | Cross-tenant or cross-connection tool aggregation |
|
|
209
|
+
| LLM agent | Interpreting intent and using bounded tools | Pagination loops, binary transfer, retry/checkpoint correctness |
|
|
210
|
+
|
|
211
|
+
## OAuth and connection profiles
|
|
212
|
+
|
|
213
|
+
### Required profiles
|
|
214
|
+
|
|
215
|
+
| Profile | Drive scope | Capability | Release posture |
|
|
216
|
+
|---|---|---|---|
|
|
217
|
+
| `selected_files` | `https://www.googleapis.com/auth/drive.file` | Files created by/opened with/explicitly selected for the app; Picker-compatible read/write | Optional limited-access product mode |
|
|
218
|
+
| `full_read` | `https://www.googleapis.com/auth/drive.readonly` | View and download all files the user can access | Default for this bulk milestone |
|
|
219
|
+
| `full_read_write` | `https://www.googleapis.com/auth/drive` | Arbitrary accessible-file mutations | Later, restricted, separate approval and action gates |
|
|
220
|
+
|
|
221
|
+
`drive.metadata.readonly` is insufficient because it cannot download content. `drive.file` is intentionally not advertised as whole-Drive access.
|
|
222
|
+
|
|
223
|
+
Gate 1 Nango authorization requests the exact normalized set `openid`, `profile`, `email`, and `https://www.googleapis.com/auth/drive.readonly` (subject to Google's current identity-scope naming). No union of historical scopes is accepted implicitly.
|
|
224
|
+
|
|
225
|
+
Full Drive scopes are restricted. If MCP Scraper stores or transmits restricted-scope data server-side, Google verification and a security assessment may be required. Release status must show this as an external gate, not silently downgrade the scope.
|
|
226
|
+
|
|
227
|
+
### Connection record additions
|
|
228
|
+
|
|
229
|
+
Add or derive these non-secret fields:
|
|
230
|
+
|
|
231
|
+
```ts
|
|
232
|
+
interface DriveConnectionCapabilities {
|
|
233
|
+
accessProfile: 'selected_files' | 'full_read' | 'full_read_write'
|
|
234
|
+
grantedScopes: string[]
|
|
235
|
+
reconnectRequired: boolean
|
|
236
|
+
toolRevision: string | null
|
|
237
|
+
schemaRevision: string | null
|
|
238
|
+
supportsFullInventory: boolean
|
|
239
|
+
supportsContentDownload: boolean
|
|
240
|
+
supportsActions: boolean
|
|
241
|
+
lastCapabilityRefreshAt: string | null
|
|
242
|
+
}
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
Do not store access or refresh tokens in scheduler/main databases.
|
|
246
|
+
|
|
247
|
+
Persist the non-secret state on `mem_external_connections` (or a versioned generic `capabilities JSONB` with equivalent indexes):
|
|
248
|
+
|
|
249
|
+
```sql
|
|
250
|
+
ALTER TABLE mem_external_connections
|
|
251
|
+
ADD COLUMN IF NOT EXISTS transport TEXT NOT NULL DEFAULT 'nango',
|
|
252
|
+
ADD COLUMN IF NOT EXISTS access_profile TEXT,
|
|
253
|
+
ADD COLUMN IF NOT EXISTS granted_scopes TEXT[] NOT NULL DEFAULT '{}',
|
|
254
|
+
ADD COLUMN IF NOT EXISTS scope_hash TEXT,
|
|
255
|
+
ADD COLUMN IF NOT EXISTS reconnect_required BOOLEAN NOT NULL DEFAULT false,
|
|
256
|
+
ADD COLUMN IF NOT EXISTS capability_state JSONB NOT NULL DEFAULT '{}',
|
|
257
|
+
ADD COLUMN IF NOT EXISTS artifact_owner_key TEXT;
|
|
258
|
+
|
|
259
|
+
ALTER TABLE mem_external_connections
|
|
260
|
+
DROP CONSTRAINT IF EXISTS mem_external_connections_identity_provider_connection_uniq;
|
|
261
|
+
ALTER TABLE mem_external_connections
|
|
262
|
+
ADD CONSTRAINT mem_external_connections_identity_transport_provider_connection_uniq
|
|
263
|
+
UNIQUE (identity, transport, provider_config_key, connection_id);
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
Add a check constraint limiting `transport` to `nango` or `remote_mcp` using the repo's duplicate-safe migration pattern. Every existing row backfills to `nango`. Routing switches on this persisted discriminator first, never on whether a provider-specific credential row happens to exist; the preview provider key provides an additional explicit boundary.
|
|
267
|
+
|
|
268
|
+
Main gives every internal user/tenant one random immutable `artifact_owner_key` stored on the user record and copies it to scheduler connection/job state. Generate it once for existing users on first migration/use. New Drive artifacts use this stable namespace; never derive ownership from the current MCP Scraper API key, email, or OAuth credential, so API-key rotation and email changes do not orphan artifacts. It is an identifier, not a download credential.
|
|
269
|
+
|
|
270
|
+
Persist pending intent without storing the Nango session token/link:
|
|
271
|
+
|
|
272
|
+
```sql
|
|
273
|
+
CREATE TABLE IF NOT EXISTS mem_pending_connection_authorizations (
|
|
274
|
+
id TEXT PRIMARY KEY,
|
|
275
|
+
identity TEXT NOT NULL,
|
|
276
|
+
provider_config_key TEXT NOT NULL,
|
|
277
|
+
requested_access_profile TEXT NOT NULL,
|
|
278
|
+
requested_scopes TEXT[] NOT NULL,
|
|
279
|
+
nango_connection_id TEXT,
|
|
280
|
+
session_correlation_hash TEXT NOT NULL,
|
|
281
|
+
status TEXT NOT NULL,
|
|
282
|
+
expires_at TIMESTAMPTZ NOT NULL,
|
|
283
|
+
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
284
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
285
|
+
CHECK (status IN ('pending','reconciling','active','reauth_required','failed','expired'))
|
|
286
|
+
);
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
The browser receives the authorization ID and one-time Nango connect URL; only the correlation hash is durable. Reconcile requires the authenticated owner plus the authorization ID and expires/rejects stale sessions.
|
|
290
|
+
|
|
291
|
+
Gate 1 exposes only `full_read`. `selected_files` is not selectable until Picker UI, selected-file/folder grant persistence, and Nango metadata synchronization are implemented. `full_read_write` remains a later reconnect/approval path.
|
|
292
|
+
|
|
293
|
+
### Reauthorization contract
|
|
294
|
+
|
|
295
|
+
Any change to required scopes marks the scheduler connection `reauth_required`; main maps that to the existing public `needs_reauth` plus `reconnectRequired=true`. The connect session pins the requested profile and exact scopes. The current product has no Nango callback state machine, so add this exact reconciliation contract:
|
|
296
|
+
|
|
297
|
+
1. main creates the signed Nango connect/reconnect session and persists the requested profile against the pending connection authorization;
|
|
298
|
+
2. after the OAuth popup returns, the frontend calls the authenticated connection reconcile endpoint; a verified Nango auth/connection webhook may trigger the same idempotent reconcile path;
|
|
299
|
+
3. scheduler fetches the Nango connection status and a non-secret granted-scope list;
|
|
300
|
+
4. preferred scope source is a source-controlled Nango post-connection hook that copies only the OAuth response's normalized `scope` names into connection metadata; Gate 0 must prove that hook/metadata path;
|
|
301
|
+
5. if Nango cannot expose scope metadata safely, the reviewed fallback fetches credentials with a narrowly scoped backend key, extracts only `scope` in memory, immediately discards tokens, and never logs/persists the credential payload;
|
|
302
|
+
6. scheduler runs capability probes, updates the external connection row, and main returns the resolved readiness state.
|
|
303
|
+
|
|
304
|
+
After reconciliation:
|
|
305
|
+
|
|
306
|
+
1. verify Nango connection status;
|
|
307
|
+
2. verify the granted scope set;
|
|
308
|
+
3. run a bounded `about.get` and one-file metadata smoke;
|
|
309
|
+
4. refresh live tool inventory;
|
|
310
|
+
5. persist only sanitized capability state;
|
|
311
|
+
6. show `Connected` only when the selected profile is actually usable.
|
|
312
|
+
|
|
313
|
+
For `selected_files` later, the Picker response's file/folder IDs are sent to an authenticated main route, ownership/profile are checked, and the server writes only those IDs to Nango connection metadata and versioned connection/schedule configuration. Client-provided IDs never expand a full-read profile or bypass provider ACL checks.
|
|
314
|
+
|
|
315
|
+
## Source-controlled Drive tool surface
|
|
316
|
+
|
|
317
|
+
The Nango dashboard is not the source of truth. Drive functions live under `nango-integrations/google-drive/`, are imported from `nango-integrations/index.ts`, compile in CI, deploy in the release, and produce a checked-in generated capability manifest.
|
|
318
|
+
|
|
319
|
+
Names below are the public provider-tool contract. If a current Nango template uses a different name, either preserve it consistently or add an explicit alias at the adapter boundary. Policy-only names are a release failure.
|
|
320
|
+
|
|
321
|
+
### Phase 1 safe reads
|
|
322
|
+
|
|
323
|
+
Gate 1 deploys this exact minimal set before broader parity work:
|
|
324
|
+
|
|
325
|
+
| Source file under `nango-integrations/google-drive/actions/` | Public tool | Bounded input contract | Purpose |
|
|
326
|
+
|---|---|---|---|
|
|
327
|
+
| `get-about.ts` | `get-about` | Empty object / reviewed fields mask only | User/storage/export-format capabilities |
|
|
328
|
+
| `list-shared-drives.ts` | `list-shared-drives` | `pageSize<=100`, opaque `pageToken` | Page shared drives |
|
|
329
|
+
| `list-drive-files.ts` | `list-drive-files` | `q`, `corpora`, optional `driveId`, `pageSize<=100`, opaque `pageToken`, bounded `orderBy` | Page normalized file metadata for an agent |
|
|
330
|
+
| `get-file-metadata.ts` | `get-file-metadata` | Required `fileId`, reviewed field set | One file/folder metadata record |
|
|
331
|
+
| `folder-content.ts` | `folder-content` | Required `folderId`, optional `driveId`, `pageSize<=100`, opaque `pageToken` | One bounded folder page |
|
|
332
|
+
| `get-changes-start-page-token.ts` | `get-changes-start-page-token` | Optional `driveId` | Establish one corpus checkpoint |
|
|
333
|
+
| `list-drive-changes.ts` | `list-drive-changes` | Required opaque `pageToken`, optional `driveId`, `pageSize<=100`, `includeRemoved=true`, `includeCorpusRemovals=true` | Page normalized changes/removals |
|
|
334
|
+
| `fetch-google-doc.ts` | `fetch-google-doc` | Required `fileId`, `maxChars<=65,536` | Bounded normalized Doc text |
|
|
335
|
+
| `fetch-google-sheet.ts` | `fetch-google-sheet` | Required `fileId`, optional `sheetName`, bounded rows/cells/bytes | Export XLSX/CSV through Drive, parse locally, and return bounded normalized Sheet data; no cell-range promise in v1 |
|
|
336
|
+
| `fetch-document.ts` | `fetch-document` | Required `fileId`, `maxBytes<=65,536` | Bounded small-file content; never base64 large data |
|
|
337
|
+
|
|
338
|
+
Each file exports its Nango action definition plus explicit Zod/input/output models, is imported from the provider index, and has a dry-run fixture. `nango-integrations/index.ts` imports the provider barrel. `nango-integrations/google-drive/models.ts` contains shared normalized File/Change/page envelopes. Exact Google endpoints, fields, required scopes, and safe error codes are documented beside each action.
|
|
339
|
+
|
|
340
|
+
`fetch-google-sheet` uses only Drive export/download in v1 and a reviewed bounded/streaming workbook parser in scheduler/Nango function code. If the exported workbook exceeds the parser's safe byte/cell budget, return a typed normalized-content-unavailable result while the archive export can still proceed through the bulk job. A future true cell-range tool must explicitly enable the Google Sheets API and scope it; it is not implied here.
|
|
341
|
+
|
|
342
|
+
Later safe-read expansion, after the minimal set and bulk path are proven: `find-file`, `get-shared-drive`, permission get/list, revision get/list, and comment/reply get/list. These names must not enter policy before their source functions and live schemas exist.
|
|
343
|
+
|
|
344
|
+
The deterministic bulk worker calls Drive REST through its internal Proxy adapter with internal page sizes up to 1,000; it does not loop the agent-facing 100-item tools. Internal inventory/byte-transfer functions do not become model-callable merely to make the scheduler work. The schedule grant identifies the approved Drive connection and read intent, while the deterministic adapter remains constrained by source-controlled read-only provider policy.
|
|
345
|
+
|
|
346
|
+
### Gated writes
|
|
347
|
+
|
|
348
|
+
Writes are classified from a source-controlled manifest, hidden from connection `tools/list` until `actions_enabled=true`, and require an exact schedule grant in scheduled runs.
|
|
349
|
+
|
|
350
|
+
| Tier | Candidate operations | Initial policy |
|
|
351
|
+
|---|---|---|
|
|
352
|
+
| Additive | upload document from an MCP Scraper artifact, create folder, copy file, create comment/reply | Enable after scope/schema/E2E review |
|
|
353
|
+
| Mutating | move file, update metadata/content, update comment/reply, update permission, hide/unhide shared drive | Account action switch + exact grant + audit |
|
|
354
|
+
| Destructive/high impact | delete/trash file, empty trash, delete permission/comment/reply/shared drive, ownership transfer, public-link mutation | Hard-block from schedules in v1; separate reviewed rollout |
|
|
355
|
+
|
|
356
|
+
Binary upload arguments use an owner-checked `artifactId` or authenticated upload handle. They never accept an arbitrary server path, raw provider URL, or unbounded base64 string.
|
|
357
|
+
|
|
358
|
+
The first write candidate is exactly `upload-document` in `actions/upload-document.ts`, accepting an owner-checked artifact/upload handle plus bounded destination/name/MIME metadata. Broader rows above are a roadmap, not Gate 1 inventory. Before any Drive write ships, replace the current generic action audit behavior with a provider-neutral structural redactor (secret/token/body/content/binary fields removed or hashed) and a safe provider-error mapper; raw action inputs and raw Google/Nango error messages must not enter `args_redacted` or public errors.
|
|
359
|
+
|
|
360
|
+
### Capability manifest
|
|
361
|
+
|
|
362
|
+
Generate a manifest with, at minimum:
|
|
363
|
+
|
|
364
|
+
```json
|
|
365
|
+
{
|
|
366
|
+
"providerConfigKey": "google-drive",
|
|
367
|
+
"generatedAt": "RFC3339",
|
|
368
|
+
"liveRevision": "sha256:...",
|
|
369
|
+
"tools": [
|
|
370
|
+
{
|
|
371
|
+
"name": "list-drive-files",
|
|
372
|
+
"classification": "read",
|
|
373
|
+
"scheduleEligible": true,
|
|
374
|
+
"destructive": false,
|
|
375
|
+
"requiredAccessProfiles": ["full_read", "full_read_write"],
|
|
376
|
+
"inputSchemaHash": "sha256:...",
|
|
377
|
+
"outputSchemaHash": "sha256:..."
|
|
378
|
+
}
|
|
379
|
+
]
|
|
380
|
+
}
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
CI and production smoke compare this with Nango live `tools/list`. Unknown live tools and policy-only tools remain unavailable and fail the release report.
|
|
384
|
+
|
|
385
|
+
## Drive inventory and reconciliation
|
|
386
|
+
|
|
387
|
+
### Selection model
|
|
388
|
+
|
|
389
|
+
```ts
|
|
390
|
+
type DriveSelection =
|
|
391
|
+
| { kind: 'all_accessible' }
|
|
392
|
+
| { kind: 'my_drive' }
|
|
393
|
+
| { kind: 'shared_drives'; driveIds: string[] }
|
|
394
|
+
| { kind: 'folders'; folderIds: string[]; followShortcuts?: boolean }
|
|
395
|
+
| { kind: 'files'; fileIds: string[] }
|
|
396
|
+
| { kind: 'query'; q: string; driveIds?: string[] }
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
`all_accessible` requires a full-read/full-write profile. Picker-scoped connections accept only file/folder selections already granted to the app.
|
|
400
|
+
|
|
401
|
+
### Full inventory
|
|
402
|
+
|
|
403
|
+
For a whole-Drive initial scan:
|
|
404
|
+
|
|
405
|
+
1. fully paginate `drives.list` and freeze the selected shared-drive ID set for this cycle;
|
|
406
|
+
2. create one corpus entry for the user and one for each selected shared drive;
|
|
407
|
+
3. obtain `changes.getStartPageToken` for the user corpus and separately with `driveId` for every shared-drive corpus before inventorying that corpus;
|
|
408
|
+
4. enumerate the user corpus with `trashed=false` and `includeItemsFromAllDrives=false` to reduce shared-drive overlap;
|
|
409
|
+
5. enumerate each shared drive with `corpora=drive`, `driveId`, `supportsAllDrives=true`, and `includeItemsFromAllDrives=true`;
|
|
410
|
+
6. follow every opaque `nextPageToken`;
|
|
411
|
+
7. if `incompleteSearch=true`, narrow the corpus rather than accepting an incomplete result;
|
|
412
|
+
8. replay `changes.list` separately for each corpus from its saved token with `includeCorpusRemovals=true` and the appropriate `driveId`/all-drives flags;
|
|
413
|
+
9. persist each corpus's `newStartPageToken` only after the final change page for that corpus succeeds.
|
|
414
|
+
|
|
415
|
+
Use `pageSize=1000` for internal REST enumeration where Google permits it. Agent-facing `list-drive-files` stays smaller (default 50, maximum 100) to bound model output.
|
|
416
|
+
|
|
417
|
+
The user corpus and per-shared-drive corpora can overlap. Deduplicate by stable file ID plus provider version while retaining the corpus/drive provenance used to discover the item. Maintain a change-token checkpoint for the user corpus and for each shared drive that participates in the selection.
|
|
418
|
+
|
|
419
|
+
Request explicit fields rather than the default partial File response. Minimum manifest fields:
|
|
420
|
+
|
|
421
|
+
```text
|
|
422
|
+
id,name,mimeType,parents,driveId,createdTime,modifiedTime,
|
|
423
|
+
size,version,headRevisionId,resourceKey,
|
|
424
|
+
md5Checksum,sha1Checksum,sha256Checksum,
|
|
425
|
+
capabilities(canDownload,canReadRevisions),
|
|
426
|
+
shortcutDetails(targetId,targetMimeType),trashed
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
Folder traversal uses bounded breadth-first queries and cycle detection. Shortcuts are metadata by default; following targets requires `followShortcuts=true` and a visited-ID set.
|
|
430
|
+
|
|
431
|
+
### Incremental sync
|
|
432
|
+
|
|
433
|
+
Drive change tokens are authoritative. Do not implement the recurring schedule as “scan modifiedTime with a seven-day overlap forever.”
|
|
434
|
+
|
|
435
|
+
Checkpoint shape:
|
|
436
|
+
|
|
437
|
+
```ts
|
|
438
|
+
interface DriveCorpusCheckpointV1 {
|
|
439
|
+
corpusKey: 'user' | `drive:${string}`
|
|
440
|
+
driveId: string | null
|
|
441
|
+
phase: 'awaiting_start_token' | 'inventory' | 'reconcile' | 'changes' | 'complete'
|
|
442
|
+
inventoryPageToken: string | null
|
|
443
|
+
initialStartPageToken: string | null
|
|
444
|
+
changePageToken: string | null
|
|
445
|
+
committedStartPageToken: string | null
|
|
446
|
+
inventoryComplete: boolean
|
|
447
|
+
}
|
|
448
|
+
|
|
449
|
+
interface DriveSyncCheckpointV1 {
|
|
450
|
+
version: 1
|
|
451
|
+
selectionHash: string
|
|
452
|
+
sharedDriveListPageToken: string | null
|
|
453
|
+
sharedDriveListingComplete: boolean
|
|
454
|
+
activeCorpusKey: string | null
|
|
455
|
+
corpora: Record<string, DriveCorpusCheckpointV1>
|
|
456
|
+
pendingContentIds: string[] // bounded to 100; larger queues live in job-item rows
|
|
457
|
+
cycleStartedAt: string
|
|
458
|
+
lastSuccessfulAt: string | null
|
|
459
|
+
}
|
|
460
|
+
```
|
|
461
|
+
|
|
462
|
+
Tokens are opaque. Validate checkpoint type and size, but never parse or transform provider tokens.
|
|
463
|
+
|
|
464
|
+
Change records include additions, modifications, removals, and lost access. Persist corpus provenance. A corpus removal updates that provenance and becomes a global tombstone only when the item is actually deleted/lost or no selected corpus still contains it; moving between corpora must not falsely delete it.
|
|
465
|
+
|
|
466
|
+
## Content download and export policy
|
|
467
|
+
|
|
468
|
+
### User-selectable content policy
|
|
469
|
+
|
|
470
|
+
```ts
|
|
471
|
+
type DriveContentPolicy =
|
|
472
|
+
| 'manifest_only'
|
|
473
|
+
| 'metadata_only'
|
|
474
|
+
| 'text_and_workspace'
|
|
475
|
+
| 'all_downloadable'
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
Default for “last seven days” is `text_and_workspace`. Default for an explicit “download everything” request is `all_downloadable`, subject to configured limits.
|
|
479
|
+
|
|
480
|
+
`text_and_workspace` produces a normalized AI-ready representation (plain text/Markdown for Docs; bounded row/cell JSONL or CSV parsed from a Drive-exported Sheet workbook) plus metadata. Slides are archive-exported in v1 and record `normalizedContentStatus='unavailable_v1'`; normalized Slides text requires a later explicit Slides/PPTX parsing adapter and tests. `all_downloadable` may additionally preserve an archive-format binary. Do not confuse DOCX/XLSX/PPTX archive exports with normalized content an agent can reason over.
|
|
481
|
+
|
|
482
|
+
### Eligibility
|
|
483
|
+
|
|
484
|
+
Google's generative-AI eligibility filtering applies to its Developer Preview remote MCP, not automatically to direct Drive REST enumeration. For the Nango/REST bulk path, MCP Scraper applies an explicit skip policy using ACL/provider errors, `capabilities.canDownload`, download/content restrictions, trash, encryption/state metadata where available, and abuse/malware signals where surfaced. Never set an abuse acknowledgement automatically. Record a typed skip reason while retaining allowed metadata.
|
|
485
|
+
|
|
486
|
+
Provider content is always `untrustedContent=true`. It is data, never instructions for the scheduler or agent.
|
|
487
|
+
|
|
488
|
+
### Ordinary blob files
|
|
489
|
+
|
|
490
|
+
Use Drive `files.get?alt=media&supportsAllDrives=true` through Nango Proxy and stream the response directly to private Blob.
|
|
491
|
+
|
|
492
|
+
- Preserve content type, content length, ETag, revision/checksum metadata.
|
|
493
|
+
- Bind every transfer checkpoint to immutable source identity: file ID plus version/head revision/checksum (or a canonical fallback version key).
|
|
494
|
+
- Use byte `Range` and a persisted offset only when the response is `206` and `Content-Range` exactly matches the requested source interval. A `200` response restarts from byte zero; never append it to prior parts.
|
|
495
|
+
- Follow only allowlisted Google/Nango redirect hosts.
|
|
496
|
+
- Apply bounded adaptive concurrency and exponential backoff for 403/429/5xx.
|
|
497
|
+
- Verify a provider checksum when available and always compute the stored SHA-256.
|
|
498
|
+
|
|
499
|
+
When a file cannot reliably finish inside one Vercel invocation, use an application-managed multipart transfer rather than one long `put()` call:
|
|
500
|
+
|
|
501
|
+
1. create a private Blob multipart upload and persist its upload ID/path;
|
|
502
|
+
2. request a bounded Drive byte range through Nango Proxy;
|
|
503
|
+
3. upload that range as one Blob part;
|
|
504
|
+
4. persist part number, ETag, next source offset, bytes, and a per-part SHA-256;
|
|
505
|
+
5. renew the job lease and continue in another background invocation;
|
|
506
|
+
6. complete the multipart upload only after every expected byte is present;
|
|
507
|
+
7. finalize a reviewed serializable incremental SHA-256 state that was checkpointed after each sequential source part, and compare provider checksums where available; a final Blob reread is allowed only for objects proven to fit one invocation;
|
|
508
|
+
8. abort the multipart upload on terminal cancellation/expiry.
|
|
509
|
+
|
|
510
|
+
The job item is not `complete` until Blob completion and checksum validation succeed. A crashed invocation can replay the current part idempotently without creating a second visible artifact.
|
|
511
|
+
|
|
512
|
+
### Google Workspace-native files
|
|
513
|
+
|
|
514
|
+
Choose a deterministic default export format, overridable per request:
|
|
515
|
+
|
|
516
|
+
| Source type | Default export | Alternate |
|
|
517
|
+
|---|---|---|
|
|
518
|
+
| Google Docs | DOCX | PDF, Markdown, plain text |
|
|
519
|
+
| Google Sheets | XLSX | PDF, CSV/TSV first sheet |
|
|
520
|
+
| Google Slides | PPTX | PDF, image first slide |
|
|
521
|
+
| Google Drawings | PDF | PNG, SVG |
|
|
522
|
+
| Apps Script | JSON | — |
|
|
523
|
+
| Google Forms | ZIP through long-running download | Typed unsupported status if unavailable |
|
|
524
|
+
| Google Sites | `text/raw` through long-running download | Typed unsupported status if unavailable |
|
|
525
|
+
| Jamboard | PDF through long-running download | Typed unsupported status if unavailable |
|
|
526
|
+
| Google Vids | MP4 through long-running download | — |
|
|
527
|
+
|
|
528
|
+
Simple `files.export` has a 10 MB output limit and does not support Range. For larger supported native content and types such as Forms/Vids, call `POST files.download` with the file's resource key and persist the returned Operation name/expiry. If the initial Operation has `done=true`, consume its `response.downloadUri` immediately and do not poll it. Otherwise poll `operations.get` as the same user with jittered exponential backoff, forwarding the resource key, until done/error/expiry; then forward the resource key again on the final download request.
|
|
529
|
+
|
|
530
|
+
Use Range on the final long-running download URI only when the Operation reports `partialDownloadAllowed=true`. Workspace exports otherwise restart from byte zero after interruption. If the deployed project/provider path cannot complete that flow, return a typed `native_export_too_large_or_unsupported` manifest status rather than truncating silently.
|
|
531
|
+
|
|
532
|
+
### Byte boundaries
|
|
533
|
+
|
|
534
|
+
Initial defaults, configurable by environment and plan:
|
|
535
|
+
|
|
536
|
+
| Boundary | Default |
|
|
537
|
+
|---|---:|
|
|
538
|
+
| Inline normalized text per file | 64 KiB |
|
|
539
|
+
| Synchronous metadata export | Existing 240 s / 5,000 records / 50 MB aggregate bounds |
|
|
540
|
+
| Per-file v1 content cap | 5 GiB soft guard; higher by plan after proof |
|
|
541
|
+
| Blob multipart threshold | 100 MiB |
|
|
542
|
+
| Job maximum files | 100,000 default plan guard |
|
|
543
|
+
| Job maximum stored bytes | Plan-specific, explicit in planning result |
|
|
544
|
+
| Manifest retention | 7 days default |
|
|
545
|
+
| Signed download URL | 15 minutes |
|
|
546
|
+
|
|
547
|
+
The transfer design supports objects beyond the soft guard; the guard is a product/cost boundary, not an excuse to buffer. Vercel Blob supports multipart transfer and much larger objects.
|
|
548
|
+
|
|
549
|
+
## Export job contract
|
|
550
|
+
|
|
551
|
+
### Root MCP tools
|
|
552
|
+
|
|
553
|
+
Keep provider-neutral names:
|
|
554
|
+
|
|
555
|
+
1. `export_connected_service_data` — validates and starts or completes an export.
|
|
556
|
+
2. `get_connected_data_export_status` — returns durable job progress and final artifacts.
|
|
557
|
+
3. `cancel_connected_data_export` — requests cancellation; already completed artifacts remain until expiry.
|
|
558
|
+
4. `renew_connected_data_download` — renews an owner-scoped manifest or file-artifact URL.
|
|
559
|
+
|
|
560
|
+
Optional later: `plan_connected_data_export` for whole-account jobs with cost/size estimates. A bounded last-seven-days export does not require a separate planning round trip.
|
|
561
|
+
|
|
562
|
+
MCP Tasks are currently an experimental protocol capability, so the first release must not depend on client task support. If a client negotiates Tasks later, the same durable `jobId` can back an MCP Task while the explicit status/cancel tools remain the compatibility path.
|
|
563
|
+
|
|
564
|
+
Do not add one top-level tool for every Drive provider action. Provider-native actions belong on the connection MCP and behind the existing generic bridges.
|
|
565
|
+
|
|
566
|
+
Static MCP annotations:
|
|
567
|
+
|
|
568
|
+
| Tool | `readOnlyHint` | `destructiveHint` | `idempotentHint` | Note |
|
|
569
|
+
|---|---:|---:|---:|---|
|
|
570
|
+
| `export_connected_service_data` | false | false | false | Creates internal job/artifact state; request-level idempotency is enforced when a job key is supplied, but the static tool supports existing non-keyed small exports |
|
|
571
|
+
| `get_connected_data_export_status` | true | false | true | Owner-scoped status read |
|
|
572
|
+
| `cancel_connected_data_export` | false | false | true | Repeated cancellation converges; never mutates Drive |
|
|
573
|
+
| `renew_connected_data_download` | true | false | false | Mints a fresh short-lived URL without changing provider data |
|
|
574
|
+
|
|
575
|
+
Public main routes retain the existing export route and add exact status/cancel routes. Scheduler adds corresponding authenticated internal create/get/cancel routes. All three resolve identity/ownership independently; the job ID alone is never authority.
|
|
576
|
+
|
|
577
|
+
### Drive datasets
|
|
578
|
+
|
|
579
|
+
Extend the dataset union:
|
|
580
|
+
|
|
581
|
+
```ts
|
|
582
|
+
type DriveDataset =
|
|
583
|
+
| 'drive_files' // file/folder metadata and content artifact references
|
|
584
|
+
| 'drive_changes' // normalized additions/modifications/removals
|
|
585
|
+
| 'drive_snapshot' // complete bounded inventory + requested content
|
|
586
|
+
```
|
|
587
|
+
|
|
588
|
+
`auto` maps a Google Drive connection to `drive_files` for a time-bounded request and `drive_snapshot` when `selection.kind='all_accessible'` without a time filter.
|
|
589
|
+
|
|
590
|
+
### Input schema
|
|
591
|
+
|
|
592
|
+
```ts
|
|
593
|
+
interface ExportConnectedServiceDataInputV2 {
|
|
594
|
+
connectionId: string
|
|
595
|
+
dataset?: ExistingDataset | DriveDataset | 'auto'
|
|
596
|
+
lastDays?: number
|
|
597
|
+
from?: string
|
|
598
|
+
to?: string
|
|
599
|
+
selection?: DriveSelection
|
|
600
|
+
contentPolicy?: DriveContentPolicy
|
|
601
|
+
exportFormats?: Partial<Record<string, string>>
|
|
602
|
+
maxItems?: number
|
|
603
|
+
maxBytes?: string
|
|
604
|
+
delivery?: 'auto' | 'artifact'
|
|
605
|
+
idempotencyKey?: string
|
|
606
|
+
continuation?: ExistingContinuation | ConnectedExportJobContinuation
|
|
607
|
+
}
|
|
608
|
+
|
|
609
|
+
interface ConnectedExportJobContinuation {
|
|
610
|
+
kind: 'job_resume'
|
|
611
|
+
token: string
|
|
612
|
+
previousJobId: string
|
|
613
|
+
expiresAt: string
|
|
614
|
+
}
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
Rules:
|
|
618
|
+
|
|
619
|
+
- Time ranges are half-open: `modifiedTime >= from && modifiedTime < to`.
|
|
620
|
+
- `idempotencyKey` is required when the call can create a background job; duplicate keys for the same tenant/connection/request hash return the same job.
|
|
621
|
+
- `maxBytes` is a decimal string to avoid JavaScript integer loss.
|
|
622
|
+
- Existing synchronous datasets keep the current 5,000-record cap. A queued Drive job may accept a higher plan-bounded `maxItems` (100,000 by default) because it checkpoints across invocations instead of returning those records inline.
|
|
623
|
+
- Whole-Drive selection requires a full access profile.
|
|
624
|
+
- Continuations remain signed/bound to tenant, connection, dataset, selection hash, range, content policy, and provider cursor.
|
|
625
|
+
|
|
626
|
+
### Output schema
|
|
627
|
+
|
|
628
|
+
```ts
|
|
629
|
+
type ExportConnectedServiceDataOutputV2 =
|
|
630
|
+
| {
|
|
631
|
+
kind: 'inline_complete'
|
|
632
|
+
ok: true
|
|
633
|
+
exportId: string
|
|
634
|
+
status: 'complete' | 'partial'
|
|
635
|
+
providerConfigKey: string
|
|
636
|
+
dataset: string
|
|
637
|
+
range: { from: string; to: string }
|
|
638
|
+
counts: ConnectedExportCounts
|
|
639
|
+
records?: unknown[]
|
|
640
|
+
preview: unknown[]
|
|
641
|
+
manifestArtifact?: ConnectedArtifactDescriptor
|
|
642
|
+
continuation: ConnectedDataContinuation | null
|
|
643
|
+
warnings: string[]
|
|
644
|
+
untrustedContent: true
|
|
645
|
+
}
|
|
646
|
+
| {
|
|
647
|
+
kind: 'job_accepted'
|
|
648
|
+
ok: true
|
|
649
|
+
jobId: string
|
|
650
|
+
status: 'queued' | 'running'
|
|
651
|
+
requestHash: string
|
|
652
|
+
statusTool: 'get_connected_data_export_status'
|
|
653
|
+
warnings: string[]
|
|
654
|
+
untrustedContent: true
|
|
655
|
+
}
|
|
656
|
+
| {
|
|
657
|
+
kind: 'terminal_error'
|
|
658
|
+
ok: false
|
|
659
|
+
errorCode: string
|
|
660
|
+
error: string
|
|
661
|
+
retryable: boolean
|
|
662
|
+
}
|
|
663
|
+
|
|
664
|
+
interface ConnectedExportCounts {
|
|
665
|
+
pages: number
|
|
666
|
+
listed: number
|
|
667
|
+
exported: number
|
|
668
|
+
failed: number
|
|
669
|
+
filteredOutsideRange: number
|
|
670
|
+
contentFetched: number
|
|
671
|
+
contentArtifacted: number
|
|
672
|
+
contentUnavailable: number
|
|
673
|
+
bytesRead: string
|
|
674
|
+
bytesStored: string
|
|
675
|
+
googleQuotaUnits: string
|
|
676
|
+
}
|
|
677
|
+
|
|
678
|
+
interface GetConnectedDataExportStatusOutput {
|
|
679
|
+
kind: 'job_status'
|
|
680
|
+
ok: true
|
|
681
|
+
jobId: string
|
|
682
|
+
status: 'queued' | 'running' | 'complete' | 'partial' | 'failed' | 'cancelled'
|
|
683
|
+
counts: ConnectedExportCounts
|
|
684
|
+
preview: unknown[]
|
|
685
|
+
manifestArtifact?: ConnectedArtifactDescriptor
|
|
686
|
+
fileArtifacts?: ConnectedArtifactDescriptor[] // capped preview only
|
|
687
|
+
continuation?: ConnectedExportJobContinuation
|
|
688
|
+
warnings: string[]
|
|
689
|
+
retryable: boolean
|
|
690
|
+
errorCode?: string
|
|
691
|
+
untrustedContent: true
|
|
692
|
+
}
|
|
693
|
+
|
|
694
|
+
interface CancelConnectedDataExportOutput {
|
|
695
|
+
kind: 'cancel_requested' | 'already_terminal'
|
|
696
|
+
ok: true
|
|
697
|
+
jobId: string
|
|
698
|
+
status: 'queued' | 'running' | 'complete' | 'partial' | 'failed' | 'cancelled'
|
|
699
|
+
}
|
|
700
|
+
```
|
|
701
|
+
|
|
702
|
+
Large artifact lists are read from the manifest, not returned as an unbounded array. A typed synchronous continuation resumes the bounded collector; a typed `job_resume` continuation starts a new owner/request-bound job after a partial terminal limit. Neither is interchangeable with a bare background `jobId`.
|
|
703
|
+
|
|
704
|
+
Each MCP implementation has an exact input/output schema and executor/API route. When a manifest signed URL is present, the tool result should include a standard MCP `resource_link` in addition to conforming `structuredContent` and a short serialized JSON text block for older clients. Never embed binary content as an MCP resource body.
|
|
705
|
+
|
|
706
|
+
### Background job persistence
|
|
707
|
+
|
|
708
|
+
Add scheduler-owned tables (names may follow the existing `mem_` convention):
|
|
709
|
+
|
|
710
|
+
```sql
|
|
711
|
+
CREATE TABLE mem_connected_export_jobs (
|
|
712
|
+
id TEXT PRIMARY KEY,
|
|
713
|
+
identity TEXT NOT NULL,
|
|
714
|
+
artifact_owner_key TEXT NOT NULL,
|
|
715
|
+
external_connection_id TEXT NOT NULL,
|
|
716
|
+
provider_config_key TEXT NOT NULL,
|
|
717
|
+
schedule_id TEXT,
|
|
718
|
+
schedule_snapshot JSONB,
|
|
719
|
+
scheduled_for TIMESTAMPTZ,
|
|
720
|
+
idempotency_key TEXT NOT NULL,
|
|
721
|
+
request_hash TEXT NOT NULL,
|
|
722
|
+
request_schema_version INTEGER NOT NULL,
|
|
723
|
+
request JSONB NOT NULL,
|
|
724
|
+
status TEXT NOT NULL,
|
|
725
|
+
checkpoint JSONB NOT NULL DEFAULT '{}',
|
|
726
|
+
checkpoint_revision BIGINT NOT NULL DEFAULT 0,
|
|
727
|
+
counts JSONB NOT NULL DEFAULT '{}',
|
|
728
|
+
entitlement_reservation JSONB NOT NULL DEFAULT '{}',
|
|
729
|
+
manifest_artifact_id TEXT,
|
|
730
|
+
error_code TEXT,
|
|
731
|
+
retryable BOOLEAN NOT NULL DEFAULT false,
|
|
732
|
+
cancel_requested_at TIMESTAMPTZ,
|
|
733
|
+
attempt INTEGER NOT NULL DEFAULT 0,
|
|
734
|
+
heartbeat_at TIMESTAMPTZ,
|
|
735
|
+
next_retry_at TIMESTAMPTZ,
|
|
736
|
+
lease_owner TEXT,
|
|
737
|
+
lease_expires_at TIMESTAMPTZ,
|
|
738
|
+
retention_expires_at TIMESTAMPTZ,
|
|
739
|
+
cleanup_state TEXT NOT NULL DEFAULT 'pending',
|
|
740
|
+
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
741
|
+
started_at TIMESTAMPTZ,
|
|
742
|
+
completed_at TIMESTAMPTZ,
|
|
743
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
744
|
+
UNIQUE (identity, external_connection_id, idempotency_key),
|
|
745
|
+
FOREIGN KEY (external_connection_id, identity)
|
|
746
|
+
REFERENCES mem_external_connections (id, identity) ON DELETE RESTRICT,
|
|
747
|
+
CHECK (status IN ('queued','running','complete','partial','failed','cancelled')),
|
|
748
|
+
CHECK (cleanup_state IN ('pending','eligible','running','complete','failed'))
|
|
749
|
+
);
|
|
750
|
+
|
|
751
|
+
CREATE TABLE mem_connected_export_items (
|
|
752
|
+
job_id TEXT NOT NULL,
|
|
753
|
+
provider_file_id TEXT NOT NULL,
|
|
754
|
+
provider_version_key TEXT NOT NULL,
|
|
755
|
+
status TEXT NOT NULL,
|
|
756
|
+
source_metadata JSONB NOT NULL,
|
|
757
|
+
artifact_id TEXT,
|
|
758
|
+
bytes_stored BIGINT,
|
|
759
|
+
sha256 TEXT,
|
|
760
|
+
retry_count INTEGER NOT NULL DEFAULT 0,
|
|
761
|
+
error_code TEXT,
|
|
762
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
763
|
+
PRIMARY KEY (job_id, provider_file_id, provider_version_key),
|
|
764
|
+
FOREIGN KEY (job_id) REFERENCES mem_connected_export_jobs (id) ON DELETE CASCADE,
|
|
765
|
+
CHECK (status IN ('pending','metadata_only','transferring','artifacted','skipped','failed','cancelled'))
|
|
766
|
+
);
|
|
767
|
+
|
|
768
|
+
CREATE TABLE mem_connected_export_transfers (
|
|
769
|
+
job_id TEXT NOT NULL,
|
|
770
|
+
provider_file_id TEXT NOT NULL,
|
|
771
|
+
provider_version_key TEXT NOT NULL,
|
|
772
|
+
source_kind TEXT NOT NULL,
|
|
773
|
+
source_operation_name TEXT,
|
|
774
|
+
source_operation_expires_at TIMESTAMPTZ,
|
|
775
|
+
partial_download_allowed BOOLEAN NOT NULL DEFAULT false,
|
|
776
|
+
source_offset BIGINT NOT NULL DEFAULT 0,
|
|
777
|
+
expected_bytes BIGINT,
|
|
778
|
+
part_size_bytes INTEGER NOT NULL,
|
|
779
|
+
blob_pathname TEXT NOT NULL,
|
|
780
|
+
blob_upload_id TEXT,
|
|
781
|
+
completed_parts JSONB NOT NULL DEFAULT '[]',
|
|
782
|
+
part_hashes JSONB NOT NULL DEFAULT '[]',
|
|
783
|
+
sha256_state BYTEA,
|
|
784
|
+
attempt INTEGER NOT NULL DEFAULT 0,
|
|
785
|
+
heartbeat_at TIMESTAMPTZ,
|
|
786
|
+
next_retry_at TIMESTAMPTZ,
|
|
787
|
+
lease_owner TEXT,
|
|
788
|
+
lease_expires_at TIMESTAMPTZ,
|
|
789
|
+
cancel_requested_at TIMESTAMPTZ,
|
|
790
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
791
|
+
PRIMARY KEY (job_id, provider_file_id, provider_version_key),
|
|
792
|
+
FOREIGN KEY (job_id, provider_file_id, provider_version_key)
|
|
793
|
+
REFERENCES mem_connected_export_items (job_id, provider_file_id, provider_version_key)
|
|
794
|
+
ON DELETE CASCADE,
|
|
795
|
+
CHECK (source_kind IN ('blob_range','workspace_export','drive_lro'))
|
|
796
|
+
);
|
|
797
|
+
|
|
798
|
+
CREATE INDEX IF NOT EXISTS mem_connected_export_jobs_claim_idx
|
|
799
|
+
ON mem_connected_export_jobs (status, next_retry_at, lease_expires_at, created_at);
|
|
800
|
+
CREATE INDEX IF NOT EXISTS mem_connected_export_jobs_owner_idx
|
|
801
|
+
ON mem_connected_export_jobs (identity, created_at DESC);
|
|
802
|
+
CREATE INDEX IF NOT EXISTS mem_connected_export_transfers_claim_idx
|
|
803
|
+
ON mem_connected_export_transfers (next_retry_at, lease_expires_at, updated_at);
|
|
804
|
+
```
|
|
805
|
+
|
|
806
|
+
Credentials, signed URLs, file bodies, and raw provider error payloads do not enter these tables.
|
|
807
|
+
|
|
808
|
+
`provider_version_key` uses, in order, Drive `version`, `headRevisionId`, a provider checksum, or a canonical hash of `(modifiedTime,size,mimeType)`; the fallback is explicit so files lacking one field remain processable. A reused idempotency key with the same request hash returns the existing job. The same key with a different request hash returns HTTP 409 `idempotency_key_conflict`.
|
|
809
|
+
|
|
810
|
+
Implement schema migration and transactional repository methods, not ad hoc SQL in routes: `createOrGetJob`, `claimJobSlice`, `renewJobLease`, `upsertItem`, `checkpointTransfer`, `requestCancellation`, `finalizeJob`, and `claimCleanup`. Claims use compare-and-set status/lease predicates, heartbeat before expiry, bounded attempts, `next_retry_at`, and connection-scoped concurrency locks.
|
|
811
|
+
|
|
812
|
+
Every checkpoint transaction increments `checkpoint_revision` and uses compare-and-set `(job id, lease owner, prior checkpoint_revision)`. Event IDs use the committed revision, so duplicate or reordered events cannot ambiguously execute the same slice.
|
|
813
|
+
|
|
814
|
+
Jobs retain an immutable schedule snapshot rather than cascading with a deleted schedule. Disconnect is a soft status transition while retained jobs/artifacts exist; hard connection deletion is restricted until cleanup has removed Blob objects and eligible job rows. Item/job cascades are used only by the cleanup repository after artifact deletion succeeds, preventing orphaned private objects and lost audit state.
|
|
815
|
+
|
|
816
|
+
### Background workflow and state machine
|
|
817
|
+
|
|
818
|
+
The scheduler owns this execution path and registers it in `src/mastra/index.ts` alongside existing functions.
|
|
819
|
+
|
|
820
|
+
```text
|
|
821
|
+
main authenticated export route
|
|
822
|
+
-> scheduler POST /api/internal/connected-exports
|
|
823
|
+
-> transaction: ownership + capability + entitlement + create/get job
|
|
824
|
+
-> emit mcp-scraper/connected-export.requested { jobId }
|
|
825
|
+
-> Inngest function id connected-export-slice
|
|
826
|
+
-> claim lease and run one bounded deterministic slice
|
|
827
|
+
-> complete/partial/fail/cancel OR emit ...connected-export.continue { jobId }
|
|
828
|
+
-> finalizer streams DB item rows into private JSONL manifest
|
|
829
|
+
-> main status route returns owner-checked artifact/resource link
|
|
830
|
+
```
|
|
831
|
+
|
|
832
|
+
Exact states and transitions:
|
|
833
|
+
|
|
834
|
+
```text
|
|
835
|
+
queued -> running | cancelled
|
|
836
|
+
running -> running (checkpoint + continuation event)
|
|
837
|
+
running -> complete (all planned items terminal, manifest finalized)
|
|
838
|
+
running -> partial (bounded user limit reached with explicit remaining work)
|
|
839
|
+
running -> failed (non-retryable or retry budget exhausted)
|
|
840
|
+
running -> cancelled (cancel observed between safe units)
|
|
841
|
+
```
|
|
842
|
+
|
|
843
|
+
Rules:
|
|
844
|
+
|
|
845
|
+
- Inngest event payload contains only `jobId`; the worker reloads tenant/connection state from Postgres.
|
|
846
|
+
- Event idempotency is `connected-export:${jobId}:${checkpointRevision}`.
|
|
847
|
+
- Concurrency is one active slice per external connection and a plan-bounded tenant maximum (default four worker slices, two ad-hoc jobs).
|
|
848
|
+
- One slice reserves at most 210 seconds of work inside the 300-second function ceiling, leaving final checkpoint/event headroom.
|
|
849
|
+
- Default binary part is 32 MiB (subject to Blob multipart minimum/maximum validation in Gate 0); metadata pages may repeat within the slice until headroom.
|
|
850
|
+
- Check cancellation before every provider page, LRO poll, Range request, Blob part, and finalization batch.
|
|
851
|
+
- Retryable 403/429/5xx honors `Retry-After`, persists `next_retry_at`, and emits/schedules a continuation with jittered exponential backoff; no blocking sleep holds the Vercel invocation.
|
|
852
|
+
- Default transfer attempt cap is eight; provider-specific non-retryable codes terminate only the item when safe, while connection/auth failures pause/fail the job.
|
|
853
|
+
- Heartbeat and lease renew after each page/part. A later worker can claim an expired lease and replay only the uncommitted page/part.
|
|
854
|
+
- Cancel aborts any open multipart upload, marks unfinished items cancelled, finalizes a cancellation manifest when possible, and releases unused entitlement reservation.
|
|
855
|
+
- Job finalization queries item rows in a stable key order and streams JSONL to Blob; it never accumulates the full manifest in memory.
|
|
856
|
+
- The current synchronous `collectConnectedDataExport` remains for small existing datasets and bounded Drive metadata only. Drive background snapshot/content jobs bypass it completely.
|
|
857
|
+
- If the initial Inngest send fails after the job transaction, a one-minute recovery function re-emits eligible queued/expired-lease jobs. Duplicate delivery is harmless because job and slice claims are idempotent.
|
|
858
|
+
|
|
859
|
+
### Entitlements, abuse controls, and settlement
|
|
860
|
+
|
|
861
|
+
Ad-hoc exports do not inherit the scheduled-occurrence billing implementation. Before queueing, atomically enforce/reserve:
|
|
862
|
+
|
|
863
|
+
- maximum active jobs per tenant and per connection;
|
|
864
|
+
- plan maximum files, source bytes, stored bytes, and retention;
|
|
865
|
+
- daily/monthly storage and egress allowance;
|
|
866
|
+
- provider/API call ceiling and Google quota-unit budget;
|
|
867
|
+
- optional MCP Scraper credit estimate for transfer/storage work.
|
|
868
|
+
|
|
869
|
+
Known-size jobs reserve bytes up front. Unknown-size native exports reserve a floor and extend the reservation before crossing it; failure to extend pauses or ends the job with `entitlement_limit_reached`, never an overage surprise. Completion/cancellation reconciles actual files/bytes/calls, settles credits, and releases the unused reservation. Download/renew routes enforce egress allowance and rate limits.
|
|
870
|
+
|
|
871
|
+
`scheduledEventUnits` is MCP Scraper product billing: one claimed schedule occurrence consumes one unit through the existing `mem_connection_sync_runs` claim path. A linked export job or continuation slice never charges another scheduled event. `googleQuotaUnits` is provider usage: one occurrence can consume many units across `files.list`, `files.get`, `files.download`, and `operations.get`. Store/report them separately and never label Google quota units as scheduled events.
|
|
872
|
+
|
|
873
|
+
### Artifact retention and deduplication decision
|
|
874
|
+
|
|
875
|
+
V1 artifacts are **job-scoped and not deduplicated across jobs**. That makes manifest-child cleanup safe and avoids reference counting. The same job/item/version remains idempotent through its unique key and multipart checkpoint.
|
|
876
|
+
|
|
877
|
+
- Ad-hoc manifests and archive binaries expire after seven days by default.
|
|
878
|
+
- Scheduled sync records persist normalized metadata and plan-bounded, deterministically chunked AI-ready text/table content in Postgres/vault/table when allowed.
|
|
879
|
+
- A scheduled record's archive artifact reference is explicitly ephemeral and includes `artifactExpiresAt` plus `artifactState`.
|
|
880
|
+
- After cleanup, mark the reference `expired_rehydratable`; do not leave a seemingly live dead pointer. A later authorized read can enqueue rehydration from the source version when still available.
|
|
881
|
+
- Cross-job content-addressed deduplication and durable archives require a later reference-counted retention design and are not silently implied by the versioned pathname.
|
|
882
|
+
|
|
883
|
+
### Manifest record
|
|
884
|
+
|
|
885
|
+
Each JSONL item is self-describing:
|
|
886
|
+
|
|
887
|
+
```json
|
|
888
|
+
{
|
|
889
|
+
"type": "drive_file",
|
|
890
|
+
"schemaVersion": 1,
|
|
891
|
+
"connectionId": "opaque-public-id",
|
|
892
|
+
"providerFileId": "...",
|
|
893
|
+
"name": "Q3 plan",
|
|
894
|
+
"path": ["My Drive", "Planning", "Q3 plan"],
|
|
895
|
+
"parents": ["..."],
|
|
896
|
+
"driveId": null,
|
|
897
|
+
"sourceMimeType": "application/vnd.google-apps.document",
|
|
898
|
+
"exportMimeType": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
|
899
|
+
"createdTime": "RFC3339",
|
|
900
|
+
"modifiedTime": "RFC3339",
|
|
901
|
+
"providerVersion": "...",
|
|
902
|
+
"providerChecksums": {"md5": null, "sha1": null, "sha256": null},
|
|
903
|
+
"canDownload": true,
|
|
904
|
+
"contentStatus": "artifacted",
|
|
905
|
+
"artifact": {
|
|
906
|
+
"artifactId": "connected-data-content/...",
|
|
907
|
+
"filename": "Q3-plan.docx",
|
|
908
|
+
"contentType": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
|
|
909
|
+
"bytes": "123456",
|
|
910
|
+
"sha256": "...",
|
|
911
|
+
"expiresAt": "RFC3339"
|
|
912
|
+
},
|
|
913
|
+
"warnings": [],
|
|
914
|
+
"untrustedContent": true
|
|
915
|
+
}
|
|
916
|
+
```
|
|
917
|
+
|
|
918
|
+
The stored manifest contains artifact IDs, not reusable signed URLs. Status/renew calls mint short-lived URLs after owner verification.
|
|
919
|
+
|
|
920
|
+
## Private Blob storage
|
|
921
|
+
|
|
922
|
+
Do not force binary files through the current string-only `createConnectedDataArtifact`. Keep that helper as the legacy manifest/NDJSON path and introduce a scheduler-owned binary artifact contract exposed through the versioned internal API:
|
|
923
|
+
|
|
924
|
+
```ts
|
|
925
|
+
type ConnectedArtifactDescriptor =
|
|
926
|
+
| { kind: 'manifest'; artifactId: string; contentType: 'application/x-ndjson'; bytes: string; sha256: string; expiresAt: string }
|
|
927
|
+
| { kind: 'binary'; artifactId: string; contentType: string; filename: string; bytes: string; sha256: string; expiresAt: string }
|
|
928
|
+
|
|
929
|
+
interface ConnectedBinaryArtifactWriter {
|
|
930
|
+
beginMultipart(...): Promise<TransferHandle>
|
|
931
|
+
uploadPart(...): Promise<CompletedPart>
|
|
932
|
+
completeMultipart(...): Promise<ConnectedArtifactDescriptor>
|
|
933
|
+
abortMultipart(...): Promise<void>
|
|
934
|
+
putStream(...): Promise<ConnectedArtifactDescriptor>
|
|
935
|
+
}
|
|
936
|
+
```
|
|
937
|
+
|
|
938
|
+
The concrete cross-repo boundary is a versioned authenticated scheduler JSON API, not a pretend repo-local shared module. Scheduler alone implements the Drive binary writer, owner parsing, allowed prefixes, TTLs, signing, and cleanup. Main validates the versioned response with its own Zod contract and exposes the public facade.
|
|
939
|
+
|
|
940
|
+
Required internal scheduler operations include job create/get/cancel plus `artifact_renew` and bounded `manifest_read`. Main forwards authenticated identity, stable artifact-owner key, job ID, and artifact ID over the existing internal-control authentication; scheduler rechecks the job/owner and returns a short-lived URL or manifest window. Contract fixtures with `apiVersion: 1` run in both repos and release parity tests compare them.
|
|
941
|
+
|
|
942
|
+
Suggested prefixes:
|
|
943
|
+
|
|
944
|
+
```text
|
|
945
|
+
connected-data-exports/{ownerId}/{jobId}/manifest-....jsonl
|
|
946
|
+
connected-data-content/{ownerId}/{jobId}/{providerFileId}/{versionHash}/{safeFilename}
|
|
947
|
+
```
|
|
948
|
+
|
|
949
|
+
Requirements:
|
|
950
|
+
|
|
951
|
+
- `access: private` for every object;
|
|
952
|
+
- pathname owner is the immutable persisted `artifact_owner_key`, matched against the authenticated user's current stable key on every public access;
|
|
953
|
+
- random suffix or collision-safe content version;
|
|
954
|
+
- streaming body; never `arrayBuffer()` the entire file;
|
|
955
|
+
- automatic `multipart: true` above 100 MiB only for transfers proven to finish in one invocation; manual multipart sessions for cross-invocation transfers;
|
|
956
|
+
- SHA-256 computed while streaming;
|
|
957
|
+
- bounded filename normalization and content-type validation;
|
|
958
|
+
- renewal checks owner, retention, and artifact/job status;
|
|
959
|
+
- cleanup deletes manifest and child content artifacts;
|
|
960
|
+
- hosted mode fails closed when private Blob is not configured.
|
|
961
|
+
|
|
962
|
+
The existing artifact reader remains manifest/text-only. Refactor its hosted implementation to issue bounded Blob range reads instead of buffering the complete artifact before slicing. Binary artifacts are delivered through signed downloads/resource links and must never enter the UTF-8 window reader. Owner, expiry, renewal, and cleanup explicitly accept both approved prefixes.
|
|
963
|
+
|
|
964
|
+
Existing short-lived artifacts created under the legacy API-key-derived prefix remain readable by the legacy path until their seven-day TTL and are not renamed. Every new Drive job uses the stable owner key contract.
|
|
965
|
+
|
|
966
|
+
## Scheduled Drive sync
|
|
967
|
+
|
|
968
|
+
### Policy
|
|
969
|
+
|
|
970
|
+
Add Drive to `CONNECTION_SYNC_REQUIRED_TOOLS` with deterministic read primitives only. Do not include `upload-document` or any mutation.
|
|
971
|
+
|
|
972
|
+
Recommended logical requirements:
|
|
973
|
+
|
|
974
|
+
```ts
|
|
975
|
+
'google-drive': [
|
|
976
|
+
'list-drive-files',
|
|
977
|
+
'list-shared-drives',
|
|
978
|
+
'get-changes-start-page-token',
|
|
979
|
+
'list-drive-changes'
|
|
980
|
+
]
|
|
981
|
+
```
|
|
982
|
+
|
|
983
|
+
Content transfer is an internal adapter capability checked separately; it is not an agent permission masquerading as a tool grant.
|
|
984
|
+
|
|
985
|
+
### Execution
|
|
986
|
+
|
|
987
|
+
1. Claim the unique `(identity, scheduleId, scheduledFor)` occurrence.
|
|
988
|
+
2. Consume one MCP Scraper `scheduledEventUnit` atomically through the existing occurrence claim.
|
|
989
|
+
3. Acquire a per-connection advisory/lease lock.
|
|
990
|
+
4. Load and validate the Drive checkpoint.
|
|
991
|
+
5. Run a fair bounded slice within the existing 240-second sync budget and 300-second Vercel ceiling.
|
|
992
|
+
6. Persist raw metadata/content artifact references and checkpoint before best-effort vault/table mirroring.
|
|
993
|
+
7. Mark `ok`, `partial`, or `failed` with per-connection totals.
|
|
994
|
+
8. Resume on the next occurrence or safe continuation task without charging the same occurrence twice.
|
|
995
|
+
|
|
996
|
+
### Stored records
|
|
997
|
+
|
|
998
|
+
Use `drive_file`, `drive_folder`, and `drive_content_chunk` record types. Fetch/store content only when the source version, head revision, or checksum changes. AI-ready text/table content larger than one record is deterministically chunked (default 64 KiB UTF-8 chunks) with file/version/chunk index and content hashes so it can persist without one giant JSONB payload. Database payloads hold normalized metadata, normalized chunks, and artifact references; archive/binary bodies remain in Blob.
|
|
999
|
+
|
|
1000
|
+
### Schedule configuration
|
|
1001
|
+
|
|
1002
|
+
```ts
|
|
1003
|
+
interface DriveScheduleConfig {
|
|
1004
|
+
selection: DriveSelection
|
|
1005
|
+
contentPolicy: DriveContentPolicy
|
|
1006
|
+
exportFormats?: Partial<Record<string, string>>
|
|
1007
|
+
maxFilesPerOccurrence: number
|
|
1008
|
+
maxBytesPerOccurrence: string
|
|
1009
|
+
mirrorToVault: boolean
|
|
1010
|
+
mirrorToTable: boolean
|
|
1011
|
+
}
|
|
1012
|
+
```
|
|
1013
|
+
|
|
1014
|
+
Persist this on the exact schedule-connection binding:
|
|
1015
|
+
|
|
1016
|
+
```sql
|
|
1017
|
+
ALTER TABLE mem_scheduled_action_connections
|
|
1018
|
+
ADD COLUMN IF NOT EXISTS provider_config JSONB NOT NULL DEFAULT '{}',
|
|
1019
|
+
ADD COLUMN IF NOT EXISTS provider_config_version INTEGER NOT NULL DEFAULT 1,
|
|
1020
|
+
ADD COLUMN IF NOT EXISTS provider_config_hash TEXT;
|
|
1021
|
+
```
|
|
1022
|
+
|
|
1023
|
+
`set_scheduled_action_connections` and the dashboard binding API accept a provider-discriminated config, validate it server-side, canonicalize/hash it, and return the resolved config. A selection/profile/content-policy change that invalidates an existing Drive checkpoint requires explicit confirmation, clears or archives the old checkpoint transactionally, and starts a new inventory cycle. Unknown config fields fail closed.
|
|
1024
|
+
|
|
1025
|
+
The UI must state that one due occurrence counts as one MCP Scraper scheduled event even when it performs many internal calls/Google quota units.
|
|
1026
|
+
|
|
1027
|
+
## MCP-native connection tool projection
|
|
1028
|
+
|
|
1029
|
+
### Protocol endpoint
|
|
1030
|
+
|
|
1031
|
+
Add this exact v1 tenant-authenticated endpoint:
|
|
1032
|
+
|
|
1033
|
+
```text
|
|
1034
|
+
POST https://mcpscraper.dev/mcp/connections/{connectionId}
|
|
1035
|
+
```
|
|
1036
|
+
|
|
1037
|
+
Add a parameterized Hono route before the existing root `/mcp` handler. Export/reuse the root caller-auth resolver, resolve the tenant-owned connection, build a per-request dynamic MCP server/tool catalog, and then hand the JSON-RPC body to the stateless Streamable HTTP transport. The route verifies ownership on every request and never trusts a prior list response.
|
|
1038
|
+
|
|
1039
|
+
V1 deliberately matches the current stateless root transport (`sessionIdGenerator: undefined`). `GET` and `DELETE` return 405. The server does **not** advertise `capabilities.tools.listChanged` and does not promise asynchronous notifications. Tool/action/reconnect changes produce a new catalog revision on the next request; revision-bound cursors from the old catalog fail with a refresh-required error. Stateful session routing and `notifications/tools/list_changed` are a later phase.
|
|
1040
|
+
|
|
1041
|
+
The endpoint supports:
|
|
1042
|
+
|
|
1043
|
+
- `initialize`;
|
|
1044
|
+
- `tools/list` with opaque cursor pagination;
|
|
1045
|
+
- the client-sent `notifications/initialized` notification;
|
|
1046
|
+
- `tools/call` with exact policy/action enforcement.
|
|
1047
|
+
|
|
1048
|
+
The conformance sequence is `initialize -> client sends notifications/initialized -> tools/list -> tools/call`. The endpoint uses the canonical `https://mcpscraper.dev/mcp` protected-resource metadata and token audience; it is an authenticated subresource, not a new OAuth audience. Its 401 challenge points to the same protected-resource metadata. Do not put credentials in the URL. The `connectionId` is safe routing input only after authentication/ownership validation. An opaque signed handle may replace it later without changing the protocol contract.
|
|
1049
|
+
|
|
1050
|
+
Apply one shared Streamable HTTP protocol guard to root and connection endpoints:
|
|
1051
|
+
|
|
1052
|
+
- if `Origin` is present, parse it and require an exact configured trusted origin; reject invalid/untrusted origins with 403; authenticated non-browser clients may omit Origin, but wildcard acceptance is forbidden;
|
|
1053
|
+
- require `Accept` to include both `application/json` and `text/event-stream`, otherwise 406;
|
|
1054
|
+
- validate `MCP-Protocol-Version` on non-initialize requests against the supported/negotiated version set, otherwise return the protocol error/400 required by the SDK contract;
|
|
1055
|
+
- a valid JSON-RPC notification, including client `notifications/initialized`, returns HTTP 202 with no body;
|
|
1056
|
+
- enforce content type/body/size and the existing auth guard before creating the per-request MCP server.
|
|
1057
|
+
|
|
1058
|
+
### Upstream catalog adapter
|
|
1059
|
+
|
|
1060
|
+
Define a provider-neutral interface:
|
|
1061
|
+
|
|
1062
|
+
```ts
|
|
1063
|
+
interface ConnectedMcpCatalogAdapter {
|
|
1064
|
+
listTools(input: {
|
|
1065
|
+
identity: string
|
|
1066
|
+
connectionId: string
|
|
1067
|
+
cursor?: string
|
|
1068
|
+
}): Promise<{
|
|
1069
|
+
protocolVersion: string | null
|
|
1070
|
+
tools: ProviderToolDefinition[]
|
|
1071
|
+
nextCursor: string | null
|
|
1072
|
+
upstreamRevision: string
|
|
1073
|
+
}>
|
|
1074
|
+
|
|
1075
|
+
callTool(input: {
|
|
1076
|
+
identity: string
|
|
1077
|
+
connectionId: string
|
|
1078
|
+
name: string
|
|
1079
|
+
arguments: Record<string, unknown>
|
|
1080
|
+
}): Promise<CallToolResult>
|
|
1081
|
+
}
|
|
1082
|
+
```
|
|
1083
|
+
|
|
1084
|
+
Implementations:
|
|
1085
|
+
|
|
1086
|
+
- `NangoConnectedMcpCatalogAdapter`: uses a low-level `@modelcontextprotocol/sdk` Client/Streamable HTTP transport with the same tenant-scoped Nango provider/connection headers, initializes once for the request/cache refresh, calls raw `listTools({cursor})`, and follows every upstream cursor.
|
|
1087
|
+
- `RemoteMcpCatalogAdapter`: uses the same low-level client against the stored tenant-scoped remote MCP OAuth connection, including Google Drive preview when enabled.
|
|
1088
|
+
|
|
1089
|
+
Do not implement passthrough through `MCPClient.listToolsetsWithErrors()` alone: the current Mastra abstraction converts schemas into tool handles and can discard raw title/icons/output-schema details and upstream pagination. Deep-clone the raw MCP Tool JSON before any Mastra conversion, then sanitize that clone. Never infer tool schemas from a hand-written TypeScript type when the live upstream MCP exposes them.
|
|
1090
|
+
|
|
1091
|
+
`upstreamRevision` is not expected from MCP. The gateway computes it as SHA-256 over a canonical serialization of the complete sanitized upstream Tool catalog in deterministic tool-name order. The downstream catalog revision is a second hash over `upstreamRevision`, source-controlled policy revision, connection/reconnect generation, transport, and current action-switch state. Cursors and caches bind to that downstream revision.
|
|
1092
|
+
|
|
1093
|
+
Extend each connection returned by `list_service_connections` with non-secret discovery metadata:
|
|
1094
|
+
|
|
1095
|
+
```ts
|
|
1096
|
+
{
|
|
1097
|
+
mcpEndpoint: string | null
|
|
1098
|
+
schemaDiscovery: 'connection_tools_list' | 'compatibility_describe'
|
|
1099
|
+
toolRevision: string | null
|
|
1100
|
+
}
|
|
1101
|
+
```
|
|
1102
|
+
|
|
1103
|
+
The frontend may offer “Copy connection MCP URL,” and an agent can use the same endpoint value without constructing it. Authentication is still required; possession of the URL grants nothing.
|
|
1104
|
+
|
|
1105
|
+
### Downstream Tool projection
|
|
1106
|
+
|
|
1107
|
+
Preserve these standard Tool fields when valid:
|
|
1108
|
+
|
|
1109
|
+
```ts
|
|
1110
|
+
interface SanitizedConnectedTool {
|
|
1111
|
+
name: string
|
|
1112
|
+
title?: string
|
|
1113
|
+
description?: string
|
|
1114
|
+
inputSchema: Record<string, unknown>
|
|
1115
|
+
outputSchema?: Record<string, unknown>
|
|
1116
|
+
annotations?: {
|
|
1117
|
+
title?: string
|
|
1118
|
+
readOnlyHint?: boolean
|
|
1119
|
+
destructiveHint?: boolean
|
|
1120
|
+
idempotentHint?: boolean
|
|
1121
|
+
openWorldHint?: boolean
|
|
1122
|
+
}
|
|
1123
|
+
icons?: Array<{ src: string; mimeType?: string; sizes?: string[]; theme?: 'light' | 'dark' }>
|
|
1124
|
+
execution?: { taskSupport: 'forbidden' | 'optional' | 'required' }
|
|
1125
|
+
}
|
|
1126
|
+
```
|
|
1127
|
+
|
|
1128
|
+
Do not forward arbitrary `_meta`, credential hints, upstream session IDs, auth challenges, or executable content. Unknown extensions are stripped unless explicitly reviewed. Preserve `execution.taskSupport` only when the gateway implements equivalent task semantics. V1 rejects a tool with `taskSupport='required'`; it may expose an optional task tool only as `forbidden` with an explicit compatibility warning or after implementing the optional path.
|
|
1129
|
+
|
|
1130
|
+
When an exposed Tool includes `outputSchema`, its `tools/call` response must return conforming `structuredContent` and should also include serialized JSON in a TextContent block for older clients. Schema passthrough without output conformance is a release failure.
|
|
1131
|
+
|
|
1132
|
+
Tool names are unchanged on a one-connection endpoint. If a future aggregate endpoint combines connections, namespace names deterministically and retain the upstream name internally; do not solve a future collision problem in the first endpoint.
|
|
1133
|
+
|
|
1134
|
+
### Callability behavior
|
|
1135
|
+
|
|
1136
|
+
- Safe reads in the live-policy intersection appear.
|
|
1137
|
+
- Actions appear only while `actions_enabled=true`.
|
|
1138
|
+
- Admin-blocked tools never appear.
|
|
1139
|
+
- Changing the action switch, reconnecting, deploying provider tools, or changing policy invalidates the catalog revision. V1 clients observe it on the next request; a future stateful transport may add list-changed notifications.
|
|
1140
|
+
- `tools/call` repeats all checks; a prior `tools/list` result is not an authorization grant.
|
|
1141
|
+
|
|
1142
|
+
### Pagination and caching
|
|
1143
|
+
|
|
1144
|
+
- Downstream cursors are opaque, signed, connection/revision-bound, and expire with the session/cache window.
|
|
1145
|
+
- Follow upstream pagination with hard page/tool/byte limits.
|
|
1146
|
+
- Positive sanitized catalog cache: at most 60 seconds.
|
|
1147
|
+
- Transient negative cache: at most 5 seconds.
|
|
1148
|
+
- Always re-check tenant ownership, connection status, action switch, and policy on a cache hit.
|
|
1149
|
+
- Cache key includes transport, external connection row ID, provider key, reconnect generation/`updatedAt`, and upstream revision.
|
|
1150
|
+
- Never cache clients, tokens, foreign-connection decisions, or raw `_meta`.
|
|
1151
|
+
|
|
1152
|
+
## Compatibility schema-description tool
|
|
1153
|
+
|
|
1154
|
+
Keep `describe_service_connection_tool` because the root MCP intentionally exposes stable generic bridges rather than every provider tool.
|
|
1155
|
+
|
|
1156
|
+
### Input
|
|
1157
|
+
|
|
1158
|
+
```ts
|
|
1159
|
+
interface DescribeServiceConnectionToolInputV2 {
|
|
1160
|
+
connectionId: string
|
|
1161
|
+
tool: string
|
|
1162
|
+
fresh?: boolean
|
|
1163
|
+
}
|
|
1164
|
+
```
|
|
1165
|
+
|
|
1166
|
+
### Output
|
|
1167
|
+
|
|
1168
|
+
```ts
|
|
1169
|
+
interface DescribeServiceConnectionToolOutputV2 {
|
|
1170
|
+
ok: boolean
|
|
1171
|
+
tool?: {
|
|
1172
|
+
name: string
|
|
1173
|
+
title: string | null
|
|
1174
|
+
description: string | null
|
|
1175
|
+
classification: 'read' | 'action'
|
|
1176
|
+
callable: boolean
|
|
1177
|
+
blockedReason: 'actions_disabled' | 'inactive_connection' | null
|
|
1178
|
+
transport: 'nango' | 'remote_mcp'
|
|
1179
|
+
providerConfigKey: string
|
|
1180
|
+
protocolVersion: string | null
|
|
1181
|
+
schemaSource: 'live_tools_list'
|
|
1182
|
+
inputSchema: Record<string, unknown>
|
|
1183
|
+
outputSchema?: Record<string, unknown>
|
|
1184
|
+
annotations?: Record<string, unknown>
|
|
1185
|
+
icons?: Array<Record<string, unknown>>
|
|
1186
|
+
execution?: { taskSupport: 'forbidden' | 'optional' | 'required' }
|
|
1187
|
+
schemaHash: string
|
|
1188
|
+
fetchedAt: string
|
|
1189
|
+
}
|
|
1190
|
+
retryable?: boolean
|
|
1191
|
+
errorCode?: string
|
|
1192
|
+
error: string | null
|
|
1193
|
+
}
|
|
1194
|
+
```
|
|
1195
|
+
|
|
1196
|
+
An action may be described while actions are disabled so the UI/agent can explain what enabling actions would grant. It remains `callable=false`, and connection-native `tools/list` omits it until enabled.
|
|
1197
|
+
|
|
1198
|
+
### Scheduler route
|
|
1199
|
+
|
|
1200
|
+
Add a strict internal route beside existing Nango read/action routes:
|
|
1201
|
+
|
|
1202
|
+
```text
|
|
1203
|
+
POST /api/internal/nango/connections/actions/describe
|
|
1204
|
+
```
|
|
1205
|
+
|
|
1206
|
+
Flow:
|
|
1207
|
+
|
|
1208
|
+
1. validate identity, connection ID, and exact tool name;
|
|
1209
|
+
2. resolve an active tenant-owned connection;
|
|
1210
|
+
3. classify the name through local policy and reject collisions/unknown/admin-blocked tools;
|
|
1211
|
+
4. open the low-level connection-scoped MCP SDK client using the Nango/remote transport adapter;
|
|
1212
|
+
5. follow raw `tools/list` pagination and find the exact live Tool;
|
|
1213
|
+
6. deep-clone the raw Tool JSON and validate/sanitize that clone before any Mastra Standard Schema conversion;
|
|
1214
|
+
7. sanitize, hash, cache, disconnect, and return.
|
|
1215
|
+
|
|
1216
|
+
Main API dispatches the existing route to Resend or Nango/remote MCP rather than returning 501 for every non-Resend transport.
|
|
1217
|
+
|
|
1218
|
+
### Schema validation
|
|
1219
|
+
|
|
1220
|
+
Reject or normalize definitions that violate any of these bounds:
|
|
1221
|
+
|
|
1222
|
+
- root `inputSchema.type` must be `object`;
|
|
1223
|
+
- `outputSchema`, when present, must be an object-root schema for stable MCP compatibility;
|
|
1224
|
+
- maximum serialized sanitized Tool: 256 KiB;
|
|
1225
|
+
- maximum schema nesting depth: 32;
|
|
1226
|
+
- maximum property count: 2,000;
|
|
1227
|
+
- maximum title: 200 characters;
|
|
1228
|
+
- maximum description: 12,000 characters;
|
|
1229
|
+
- tool name follows MCP length/character guidance and exactly matches the requested live name;
|
|
1230
|
+
- icon entries are bounded HTTPS/data descriptors and are never fetched server-side during discovery;
|
|
1231
|
+
- external/remote `$ref` is rejected; local references must be resolved or preserved safely within the bounded schema;
|
|
1232
|
+
- secret-looking defaults/examples, credential values, `_meta`, and unknown auth extensions are stripped;
|
|
1233
|
+
- upstream name must exactly equal the requested name.
|
|
1234
|
+
|
|
1235
|
+
Treat provider descriptions as untrusted text. They may guide arguments, never override MCP Scraper policy.
|
|
1236
|
+
|
|
1237
|
+
### Safe error contract
|
|
1238
|
+
|
|
1239
|
+
| Status | Code | Meaning |
|
|
1240
|
+
|---:|---|---|
|
|
1241
|
+
| 400 | `tool_not_allowed` / `invalid_schema` | Name outside policy or malformed upstream schema |
|
|
1242
|
+
| 404 | `connection_not_found` | Missing or foreign connection, same public response |
|
|
1243
|
+
| 409 | `connection_inactive` | Reauthorization or repair needed |
|
|
1244
|
+
| 429 | `upstream_rate_limited` | Retryable |
|
|
1245
|
+
| 502 | `tool_discovery_failed` / `live_tool_missing` | Live inventory cannot satisfy policy |
|
|
1246
|
+
| 503 | `connection_transport_unavailable` | Nango/remote MCP/config temporarily unavailable |
|
|
1247
|
+
|
|
1248
|
+
Main control clients preserve safe upstream status/code/retryability instead of collapsing all failures to 502.
|
|
1249
|
+
|
|
1250
|
+
This requires two prerequisite error-boundary fixes. Scheduler `safeRouteError` maps provider errors to stable codes, preserves safe 429 versus 503 semantics, and never returns raw Nango/Google messages. Main `NangoControlError`/`controlRequest` accepts and safely forwards reviewed 400/404/409/429/502/503 plus `code` and `retryable` instead of converting every non-2xx response to 502/503. The `fresh` describe option is rate-limited and cannot bypass ownership/policy checks.
|
|
1251
|
+
|
|
1252
|
+
## Security and guardrails
|
|
1253
|
+
|
|
1254
|
+
### Tenant and credential isolation
|
|
1255
|
+
|
|
1256
|
+
- Resolve identity from authenticated caller only.
|
|
1257
|
+
- Query connection by `(identity, externalConnectionId)`.
|
|
1258
|
+
- Keep Nango connection IDs/provider config keys out of user-visible logs where unnecessary.
|
|
1259
|
+
- Never return OAuth codes/tokens, Nango secrets, Blob read/write/admin tokens, or standalone signing tokens. A short-lived owner-authorized presigned download URL/resource link is an intentional delivery result; it is never logged or persisted in manifests/DB and expires after the declared window.
|
|
1260
|
+
- Encrypt remote-MCP OAuth state with the existing remote credential storage.
|
|
1261
|
+
|
|
1262
|
+
### Provider/API safety
|
|
1263
|
+
|
|
1264
|
+
- Nango Proxy base URL is fixed to Google Drive API or an allowlisted Google download origin.
|
|
1265
|
+
- Do not accept arbitrary `baseUrlOverride` from a user/tool argument.
|
|
1266
|
+
- Bound redirects, timeouts, response headers, and content lengths.
|
|
1267
|
+
- Honor `Retry-After`; apply jittered truncated exponential backoff.
|
|
1268
|
+
- Check actual Cloud quotas; do not hardcode concurrency from headline limits.
|
|
1269
|
+
|
|
1270
|
+
As of this spec, Google's post-May-1-2026 model documents 1,000,000 quota units/minute/project, 325,000/minute/user/project, and 1 TB/day/project egress, while legacy-active projects may retain older quotas. Example costs include `files.get` at 5 units, `files.list` at 100, and `files.download` at 200; every `operations.get` poll also consumes provider quota. Treat these as observed planning inputs, read the deployed Cloud project's actual limits at release, and record units per operation rather than assuming one API request equals one unit.
|
|
1271
|
+
|
|
1272
|
+
### Prompt injection
|
|
1273
|
+
|
|
1274
|
+
- File text, comments, filenames, and descriptions are untrusted content.
|
|
1275
|
+
- Deterministic workers never execute instructions from provider content.
|
|
1276
|
+
- MCP output marks content untrusted and separates it from warnings/control metadata.
|
|
1277
|
+
- Scheduled agent reads retain existing injected-content defenses.
|
|
1278
|
+
|
|
1279
|
+
### Writes
|
|
1280
|
+
|
|
1281
|
+
- `actions_enabled` is false by default.
|
|
1282
|
+
- Exact action tool and arguments are audited with sensitive fields redacted.
|
|
1283
|
+
- Destructive operations are unavailable to schedules in v1.
|
|
1284
|
+
- Upload/copy/create operations use idempotency keys where the provider permits.
|
|
1285
|
+
- Permissions/public links/ownership transfer require separate product approval and clear UI copy.
|
|
1286
|
+
|
|
1287
|
+
### Storage and retention
|
|
1288
|
+
|
|
1289
|
+
- Private artifacts only.
|
|
1290
|
+
- Default seven-day retention is visible to the user and configurable by plan.
|
|
1291
|
+
- Deleting/disconnecting a connection can enqueue early artifact cleanup subject to retention/audit rules.
|
|
1292
|
+
- Blob lifecycle cleanup is idempotent and records counts/errors without filenames or content.
|
|
1293
|
+
|
|
1294
|
+
## Observability
|
|
1295
|
+
|
|
1296
|
+
### Metrics
|
|
1297
|
+
|
|
1298
|
+
Record by provider, plan, and safe outcome class:
|
|
1299
|
+
|
|
1300
|
+
- connection/reconnect success and scope mismatch;
|
|
1301
|
+
- live tool count by read/action/blocked classification;
|
|
1302
|
+
- schema discovery latency, pages, bytes, cache hit, invalid definitions;
|
|
1303
|
+
- export jobs queued/running/partial/complete/failed/cancelled;
|
|
1304
|
+
- files listed/eligible/skipped/downloaded/exported;
|
|
1305
|
+
- bytes read/stored, multipart uploads, Range resumes;
|
|
1306
|
+
- Drive/Nango status families, retry counts, rate-limit waits;
|
|
1307
|
+
- schedule occurrences, `scheduledEventUnits`, Google quota units, leased retries, checkpoint resumes;
|
|
1308
|
+
- artifact renewals/expiry/cleanup.
|
|
1309
|
+
|
|
1310
|
+
### Safe structured logs
|
|
1311
|
+
|
|
1312
|
+
Each event carries:
|
|
1313
|
+
|
|
1314
|
+
```text
|
|
1315
|
+
requestId, identityHash, connectionPublicId, providerConfigKey,
|
|
1316
|
+
scheduleId?, occurrenceId?, jobId?, toolName?, schemaHash?,
|
|
1317
|
+
statusCode?, safeErrorCode?, retryable?, durationMs,
|
|
1318
|
+
files?, bytes?, pages?, attempt?
|
|
1319
|
+
```
|
|
1320
|
+
|
|
1321
|
+
No file content, access token, full signed URL, raw Google error body, or sensitive filename is logged by default.
|
|
1322
|
+
|
|
1323
|
+
### Run receipts
|
|
1324
|
+
|
|
1325
|
+
Every scheduled Drive occurrence persists:
|
|
1326
|
+
|
|
1327
|
+
- scheduled time, start/end, status, MCP Scraper scheduled-event unit consumed, and Google quota units observed;
|
|
1328
|
+
- connection and selection hash;
|
|
1329
|
+
- checkpoint before/after revision;
|
|
1330
|
+
- pages/files/changed/tombstoned/content counts;
|
|
1331
|
+
- bytes and artifacts created;
|
|
1332
|
+
- safe warnings/errors;
|
|
1333
|
+
- continuation needed;
|
|
1334
|
+
- linked export job IDs.
|
|
1335
|
+
|
|
1336
|
+
## API, SDK, UI, and documentation surfaces
|
|
1337
|
+
|
|
1338
|
+
### Main API/MCP
|
|
1339
|
+
|
|
1340
|
+
Update:
|
|
1341
|
+
|
|
1342
|
+
- connected dataset unions and validation;
|
|
1343
|
+
- export output/job status types;
|
|
1344
|
+
- Nango/remote MCP schema description dispatch;
|
|
1345
|
+
- connection-scoped MCP route;
|
|
1346
|
+
- descriptions for `read_service_connection`, `describe_service_connection_tool`, `export_connected_service_data`, and renewal/status/cancel tools;
|
|
1347
|
+
- generated MCP manifest and parity tests.
|
|
1348
|
+
|
|
1349
|
+
### Scheduler
|
|
1350
|
+
|
|
1351
|
+
Update:
|
|
1352
|
+
|
|
1353
|
+
- Drive OAuth connect-session defaults;
|
|
1354
|
+
- source-controlled live read/action policy;
|
|
1355
|
+
- provider capability resolution;
|
|
1356
|
+
- Nango schema describe route;
|
|
1357
|
+
- Drive REST/Proxy inventory and content adapter;
|
|
1358
|
+
- Drive connected-data export page/job routes;
|
|
1359
|
+
- deterministic `connection_sync` policy/core/records/checkpoints/receipts;
|
|
1360
|
+
- export job tables and background functions.
|
|
1361
|
+
|
|
1362
|
+
The scheduler owns execution and private-Blob operations for background Drive jobs because it already has Inngest and durable run/checkpoint state. Add a dedicated private Blob token and `@vercel/blob` there. The main API remains the public authorization/delivery facade: it authenticates the caller and invokes the scheduler's owner-checking internal status/renew/manifest-read endpoints. Main does not write or sign new Drive job artifacts.
|
|
1363
|
+
|
|
1364
|
+
Do not add Drive to the legacy unmetered digest path.
|
|
1365
|
+
|
|
1366
|
+
### Implementation file map
|
|
1367
|
+
|
|
1368
|
+
Primary existing files to change or extend:
|
|
1369
|
+
|
|
1370
|
+
| Repo | Existing surface | Change |
|
|
1371
|
+
|---|---|---|
|
|
1372
|
+
| main | `src/api/server.ts` | Dispatch Nango/remote schema descriptions; Drive export/job/status/cancel routes; connection MCP mount |
|
|
1373
|
+
| main | `src/api/nango-control.ts` | Typed describe/export-job scheduler clients with safe status preservation |
|
|
1374
|
+
| main | `src/api/connected-data-export.ts` | Drive datasets and queued/running states while preserving existing bounded exporters |
|
|
1375
|
+
| main | `src/api/connected-data-artifacts.ts` | Keep legacy manifest/text path; route new Drive artifact renew/window operations to scheduler internal API |
|
|
1376
|
+
| scheduler | new connected-artifact contract + binary writer modules | Approved prefixes/TTL/types plus streaming/manual-multipart/signing/cleanup implementation |
|
|
1377
|
+
| main | `src/mcp/mcp-tool-schemas.ts` | V2 connection/schema/export/job schemas |
|
|
1378
|
+
| main | `src/mcp/paa-mcp-server.ts` | Register generic status/cancel tools and update descriptions |
|
|
1379
|
+
| main | `src/mcp/mcp-routes.ts` | Export caller auth and mount stateless `/mcp/connections/:connectionId` before the root handler |
|
|
1380
|
+
| main | `src/mcp/http-mcp-tool-executor.ts`, `src/mcp/IMcpToolExecutor.ts` | Wire new contracts |
|
|
1381
|
+
| main | `nango-integrations/index.ts`, `nango-integrations/google-drive/**` | Source-controlled Drive functions and schemas |
|
|
1382
|
+
| scheduler | `src/mastra/lib/nango-api.ts` | Explicit Drive scopes and reviewed live read/action policy |
|
|
1383
|
+
| scheduler | `src/mastra/lib/nango-connection-client.ts` | Existing Mastra execution client remains for bound agent tools |
|
|
1384
|
+
| scheduler | new raw MCP catalog adapter | Low-level SDK initialize/paginated `tools/list` with raw Tool preservation |
|
|
1385
|
+
| scheduler | `src/mastra/routes/nango-integrations.ts` | Describe, Drive page/export/job internal routes |
|
|
1386
|
+
| scheduler | Nango connection reconcile/hook modules | Requested profile, post-connection scopes, capability probes, `needs_reauth` state |
|
|
1387
|
+
| scheduler | `src/mastra/lib/connection-sync-policy.ts` | Deterministic Drive required capabilities |
|
|
1388
|
+
| scheduler | `src/mastra/lib/connection-sync-core.ts` | Drive snapshot/changes/content dispatch |
|
|
1389
|
+
| scheduler | `src/mastra/db/external-connections.ts` | Capability/schedule-config/export job/item/transfer schema and repositories |
|
|
1390
|
+
| scheduler | `src/mastra/index.ts` | Register export slice/recovery/cleanup Inngest functions and 300-second limits |
|
|
1391
|
+
| scheduler | route error/action audit helpers | Structural redaction and stable safe provider status/error mapping |
|
|
1392
|
+
| scheduler | new `src/mastra/lib/google-drive-*.ts` modules | REST/Proxy inventory, changes, content transfer, manifest normalization |
|
|
1393
|
+
| scheduler | new background workflow/function files | Job leases, multipart slices, completion/cancellation |
|
|
1394
|
+
| frontend | `public/app.jsx` and compiled asset | Scope-aware Drive card/drawer/export/schedule/MCP URL UI |
|
|
1395
|
+
| tests/docs | connection/export/release suites and generated manifest | Cross-surface contracts and production receipts |
|
|
1396
|
+
|
|
1397
|
+
The public SDK repo adds the corresponding connection/export/job types and methods; the web repo adds the marketing integrations/scheduling copy without becoming a second capability source of truth.
|
|
1398
|
+
|
|
1399
|
+
### Environment contract
|
|
1400
|
+
|
|
1401
|
+
Reference names only; never commit values:
|
|
1402
|
+
|
|
1403
|
+
| Variable | Runtime | Purpose |
|
|
1404
|
+
|---|---|---|
|
|
1405
|
+
| existing Nango secret/MCP URL variables | scheduler | MCP catalog/calls and Proxy authentication |
|
|
1406
|
+
| existing `CONNECTED_DATA_READ_WRITE_TOKEN` | main | Legacy synchronous connected-data artifacts only |
|
|
1407
|
+
| `DRIVE_EXPORT_BLOB_READ_WRITE_TOKEN` | scheduler | Dedicated private Drive job store: write, multipart, sign, bounded manifest reads, cleanup |
|
|
1408
|
+
| existing scheduler internal-control secret | main + scheduler | Authenticated job/reconcile/status/cancel routes |
|
|
1409
|
+
| existing Inngest event/signing variables | scheduler | Export slice/recovery/cleanup execution |
|
|
1410
|
+
| `GOOGLE_DRIVE_REMOTE_MCP_ENABLED` | scheduler/main | Default false preview adapter gate |
|
|
1411
|
+
| `GOOGLE_DRIVE_REMOTE_MCP_URL` | scheduler | Defaults to Google's documented endpoint only when feature enabled |
|
|
1412
|
+
| `MCP_SCRAPER_CONNECTED_EXPORT_PART_BYTES` | scheduler | Default 32 MiB after Gate 0 validation |
|
|
1413
|
+
| `MCP_SCRAPER_CONNECTED_EXPORT_SLICE_BUDGET_MS` | scheduler | Default 210,000 ms under 300-second ceiling |
|
|
1414
|
+
| plan/entitlement limits | main/scheduler | File, byte, concurrency, storage, egress, retry, retention boundaries |
|
|
1415
|
+
|
|
1416
|
+
Production startup/status fails closed for background content exports when the private store, internal auth, Inngest, or Nango Proxy prerequisites are missing. Metadata-only interactive reads may remain available and must report the narrower readiness state.
|
|
1417
|
+
|
|
1418
|
+
### Frontend
|
|
1419
|
+
|
|
1420
|
+
Integration card/drawer must show:
|
|
1421
|
+
|
|
1422
|
+
- Google Drive logo and account identity;
|
|
1423
|
+
- access profile (`Selected files`, `Full Drive read`, or `Full Drive read/write`);
|
|
1424
|
+
- real status (`Needs reconnect`, `Connected`, `Tools unavailable`, `Ready for sync`);
|
|
1425
|
+
- actual read/action counts from resolved capabilities;
|
|
1426
|
+
- action toggle only when an action-capable scope/tool exists;
|
|
1427
|
+
- reconnect reason when scopes/functions changed;
|
|
1428
|
+
- bulk export controls and recent job state;
|
|
1429
|
+
- schedule Data Sync configuration for selection/content/limits;
|
|
1430
|
+
- artifact retention and scheduled-event billing copy.
|
|
1431
|
+
- if the optional Google preview MCP is enabled, a separate preview badge and disclosure that its OAuth setup includes `drive.file` even while downstream write tools are hidden.
|
|
1432
|
+
|
|
1433
|
+
The integrations route and header/hamburger navigation must be present in production; tests assert both desktop and mobile navigation.
|
|
1434
|
+
|
|
1435
|
+
### SDK/npm
|
|
1436
|
+
|
|
1437
|
+
Add typed clients/types for:
|
|
1438
|
+
|
|
1439
|
+
- list connections;
|
|
1440
|
+
- describe connection tool;
|
|
1441
|
+
- create/list connection MCP endpoint metadata if exposed through REST;
|
|
1442
|
+
- start export;
|
|
1443
|
+
- get/cancel export job;
|
|
1444
|
+
- renew artifact URL;
|
|
1445
|
+
- Drive selection/content policy/datasets/manifests.
|
|
1446
|
+
|
|
1447
|
+
The SDK does not embed Google or Nango credentials. It calls MCP Scraper with the user's MCP Scraper auth.
|
|
1448
|
+
|
|
1449
|
+
### Documentation
|
|
1450
|
+
|
|
1451
|
+
Update README, integrations guide, scheduler guide, hosted skill, `llms.txt`/`llms-full.txt`, generated manifest, MCPB, and marketing surfaces. Clearly distinguish:
|
|
1452
|
+
|
|
1453
|
+
- OAuth connected;
|
|
1454
|
+
- interactive tools live;
|
|
1455
|
+
- bulk/scheduled sync supported;
|
|
1456
|
+
- action scope and switch;
|
|
1457
|
+
- official Google remote MCP preview versus MCP Scraper's Nango/REST bulk path.
|
|
1458
|
+
|
|
1459
|
+
## Test plan
|
|
1460
|
+
|
|
1461
|
+
### Unit and contract
|
|
1462
|
+
|
|
1463
|
+
- Drive OAuth profile -> exact scopes; changed scope -> reconnect required.
|
|
1464
|
+
- Post-connection scope metadata/reconcile path persists scopes only; tokens never reach DB/logs. Missing required scope produces `needs_reauth`.
|
|
1465
|
+
- Live function inventory intersects policy; stale policy-only tools never expose.
|
|
1466
|
+
- Read/action collision fails closed.
|
|
1467
|
+
- Admin-blocked and destructive scheduled tools never expose.
|
|
1468
|
+
- Action switch changes callability and catalog revision.
|
|
1469
|
+
- Nango describe returns exact input/output schema and safe annotations.
|
|
1470
|
+
- Description of disabled action succeeds with `callable=false`; action call fails.
|
|
1471
|
+
- Upstream second-page Tool is discoverable.
|
|
1472
|
+
- Schema root/size/depth/property/ref/secret sanitization bounds.
|
|
1473
|
+
- Cache rechecks ownership/status/policy and invalidates on reconnect/deploy/toggle.
|
|
1474
|
+
- Client disconnects on success and every error path.
|
|
1475
|
+
- Main preserves safe 400/404/409/429/502/503 errors.
|
|
1476
|
+
- Action audit redactor removes content/token/binary fields; public errors contain only safe codes.
|
|
1477
|
+
- Root MCP output schemas validate for every new result state.
|
|
1478
|
+
- Same idempotency key/same request returns one job; same key/different request returns 409.
|
|
1479
|
+
- Duplicate/reordered Inngest events and an expired worker lease do not duplicate a job, item, or Blob part.
|
|
1480
|
+
- Queued-job recovery re-emits work after an initial event-send failure.
|
|
1481
|
+
- Cancellation aborts multipart state and releases entitlement reservation.
|
|
1482
|
+
- Ad-hoc concurrency/file/byte/storage/egress limits fail before unbounded work and settle actual usage.
|
|
1483
|
+
|
|
1484
|
+
### Inventory and change sync
|
|
1485
|
+
|
|
1486
|
+
- My Drive with more than 1,000 files fully paginates.
|
|
1487
|
+
- Shared drive with more than 1,000 files fully paginates.
|
|
1488
|
+
- Multiple shared drives are isolated into corpora; `incompleteSearch` triggers narrowing.
|
|
1489
|
+
- `drives.list` itself paginates; every corpus has independent initial/current/committed change tokens.
|
|
1490
|
+
- Initial start token + inventory + reconciliation captures a mutation during crawl.
|
|
1491
|
+
- `nextPageToken` and final `newStartPageToken` commit correctly.
|
|
1492
|
+
- Removal/lost access creates a tombstone, while a corpus move only updates provenance.
|
|
1493
|
+
- Checkpoint resumes without skips or duplicates.
|
|
1494
|
+
- Non-advancing token fails partial with typed warning.
|
|
1495
|
+
- Selection change invalidates incompatible checkpoint.
|
|
1496
|
+
|
|
1497
|
+
### Content transfer
|
|
1498
|
+
|
|
1499
|
+
- Small UTF-8 text stays within inline bound.
|
|
1500
|
+
- Large text becomes an artifact.
|
|
1501
|
+
- Blob file streams through Nango Proxy without buffering or MCP base64.
|
|
1502
|
+
- Multipart is used above 100 MiB.
|
|
1503
|
+
- A matching `206`/`Content-Range` resumes the source revision; a `200` restarts from zero; source-version mismatch discards incompatible parts.
|
|
1504
|
+
- A worker stops after a committed multipart part, a new invocation resumes from persisted upload ID/parts/offset, and completion produces one valid object.
|
|
1505
|
+
- Workspace export under 10 MB succeeds.
|
|
1506
|
+
- Workspace export above 10 MB uses the exact long-running flow (including initial `done=true`, resource key, Operation expiry, and partial-download flag) or records typed unsupported status.
|
|
1507
|
+
- Sheet normalization parses only configured rows/cells/sheet name from a Drive export and fails typed when the parser budget is exceeded; no v1 cell-range claim appears in schema.
|
|
1508
|
+
- Slides return an archive artifact plus `normalizedContentStatus='unavailable_v1'` until a separately tested normalization adapter exists.
|
|
1509
|
+
- Remote-MCP eligibility-filter tests are separate from REST bulk skip-policy tests; REST `canDownload` false/restriction/encryption/trash/provider-error paths produce explicit manifest outcomes.
|
|
1510
|
+
- Content failure retains metadata.
|
|
1511
|
+
- Raw provider URL/token never appears in result, DB, or logs.
|
|
1512
|
+
- Cross-tenant artifact renew/read is denied.
|
|
1513
|
+
- 15-minute URL, seven-day expiry, and cleanup are enforced.
|
|
1514
|
+
- Cleanup marks scheduled archive references `expired_rehydratable`; no cross-job object is deleted because v1 has no cross-job deduplication.
|
|
1515
|
+
- Finalizer streams more manifest rows than fit memory and the window reader fetches only the requested Blob range.
|
|
1516
|
+
|
|
1517
|
+
### Scheduler and billing
|
|
1518
|
+
|
|
1519
|
+
- Drive binding is accepted only with exact deterministic read requirements.
|
|
1520
|
+
- Initial snapshot and incremental change modes persist records/checkpoints.
|
|
1521
|
+
- Changed-only content retrieval prevents repeated byte downloads.
|
|
1522
|
+
- 240-second budget exits partial with a durable checkpoint.
|
|
1523
|
+
- Fair connection slice prevents one Drive from starving other connections.
|
|
1524
|
+
- One scheduled occurrence consumes exactly one `scheduledEventUnit` while separately recording Google quota units.
|
|
1525
|
+
- Lease retry of the same occurrence does not double charge.
|
|
1526
|
+
- Concurrent occurrence/connection lock prevents overlapping Drive crawls.
|
|
1527
|
+
- Run receipt totals reconcile with stored records/artifacts.
|
|
1528
|
+
|
|
1529
|
+
### MCP protocol
|
|
1530
|
+
|
|
1531
|
+
- Connection endpoint `initialize -> client sends notifications/initialized -> tools/list -> tools/call` works with API key and OAuth caller.
|
|
1532
|
+
- `tools/list` returns only the selected connection's callable tools.
|
|
1533
|
+
- Read-only mode hides writes.
|
|
1534
|
+
- Every Tool has valid object-root `inputSchema`; output schemas validate structured content.
|
|
1535
|
+
- Cursor pagination is opaque and revision-bound.
|
|
1536
|
+
- V1 stateless connection/root transports do not advertise list-changed; a revision change invalidates an old cursor and the next request sees the new catalog.
|
|
1537
|
+
- Trusted/absent Origin behavior passes; an untrusted present Origin returns 403 on both root and connection endpoints.
|
|
1538
|
+
- Missing required dual Accept values returns 406; unsupported `MCP-Protocol-Version` is rejected; valid notifications return HTTP 202 with an empty body.
|
|
1539
|
+
- Compatibility describe matches the exact corresponding native Tool/schema hash.
|
|
1540
|
+
- Google's preview MCP live canary discovers its documented eight tools when feature flag and preview access are enabled.
|
|
1541
|
+
- The public preview connection hides `download_file_content`, `create_file`, and `copy_file` in read-only mode; bounded `read_file_content` enforces the gateway response cap.
|
|
1542
|
+
|
|
1543
|
+
### Production smoke
|
|
1544
|
+
|
|
1545
|
+
Using a dedicated test Drive/account:
|
|
1546
|
+
|
|
1547
|
+
1. connect full-read profile and verify granted scope;
|
|
1548
|
+
2. prove nonzero live Nango tools and exact policy intersection;
|
|
1549
|
+
3. describe `list-drive-files` through root MCP;
|
|
1550
|
+
4. list the same tool through the connection MCP;
|
|
1551
|
+
5. export a seven-day metadata corpus spanning more than one page;
|
|
1552
|
+
6. export one small Google Doc;
|
|
1553
|
+
7. stream one binary larger than the inline/action limits to private Blob;
|
|
1554
|
+
8. run one scheduled Drive occurrence and verify one scheduled-event billing unit plus separately reported Google quota units;
|
|
1555
|
+
9. renew the manifest URL as owner and reject a different tenant;
|
|
1556
|
+
10. disconnect/reconnect or toggle actions and verify catalog revision behavior.
|
|
1557
|
+
|
|
1558
|
+
## Release sequence and phase gates
|
|
1559
|
+
|
|
1560
|
+
### Gate 0 — spec and source truth
|
|
1561
|
+
|
|
1562
|
+
- This spec accepted.
|
|
1563
|
+
- Clean current release worktrees.
|
|
1564
|
+
- Live sanitized baseline saved in release receipt.
|
|
1565
|
+
- No secret material in artifacts.
|
|
1566
|
+
- Spike A proves low-level paginated Nango MCP `tools/list` capture preserves raw title, description, input/output schemas, annotations, icons, execution metadata, and cursors before Mastra conversion.
|
|
1567
|
+
- Spike B proves Nango Proxy streams Drive bodies, forwards a bounded Range, preserves `206`/`Content-Range`, handles `200` restart, and can follow or explicitly proxy a Drive LRO final download URI/resource key without credential leakage.
|
|
1568
|
+
- Spike C proves a manual private Vercel Blob multipart upload can persist upload ID/path/completed parts plus serializable incremental SHA-256 state, stop one invocation, resume/hash in another, complete once, and abort on cancellation.
|
|
1569
|
+
- Spike D proves the Nango post-connection metadata path for non-secret granted scopes, or approves the narrow in-memory credentials fallback.
|
|
1570
|
+
- Product decisions set ad-hoc file/byte/concurrency/storage/egress entitlements and confirm seven-day ephemeral archive retention.
|
|
1571
|
+
- Implementation does not begin past the affected boundary until its spike has a saved test/receipt.
|
|
1572
|
+
|
|
1573
|
+
### Gate 1 — OAuth and live read tools
|
|
1574
|
+
|
|
1575
|
+
- Google Cloud production OAuth credentials/scopes configured in Nango.
|
|
1576
|
+
- Google verification/security gate status recorded.
|
|
1577
|
+
- Source-controlled Drive functions compile/deploy.
|
|
1578
|
+
- The exact ten-tool Gate 1 read set, and no aspirational later tools, appears live.
|
|
1579
|
+
- Live catalog and policy manifest match.
|
|
1580
|
+
- Existing Drive account reauthorized and active.
|
|
1581
|
+
- Interactive safe reads work through generic root MCP and scheduled Mastra injection.
|
|
1582
|
+
|
|
1583
|
+
### Gate 2 — native schema passthrough
|
|
1584
|
+
|
|
1585
|
+
- Nango describe route and main dispatch shipped.
|
|
1586
|
+
- Connection-scoped MCP endpoint ships `tools/list`/`tools/call`.
|
|
1587
|
+
- Schema sanitization/pagination/cache/revision-invalidation/stateless transport tests pass.
|
|
1588
|
+
- Google remote MCP adapter remains feature-flagged but passes conformance test where preview access exists.
|
|
1589
|
+
|
|
1590
|
+
### Gate 3 — deterministic metadata export
|
|
1591
|
+
|
|
1592
|
+
- Drive datasets, inventory, shared-drive traversal, changes reconciliation, checkpoints, and manifests pass tests.
|
|
1593
|
+
- `export_connected_service_data(lastDays=7)` no longer loops provider reads through the model.
|
|
1594
|
+
- Main/scheduler contracts agree.
|
|
1595
|
+
|
|
1596
|
+
### Gate 4 — large content and private artifacts
|
|
1597
|
+
|
|
1598
|
+
- Stream/multipart/Range/LRO paths validated.
|
|
1599
|
+
- A transfer is interrupted after a committed part and resumes in another worker invocation without duplicated/corrupted bytes.
|
|
1600
|
+
- Per-file and manifest owner isolation/retention/renewal/cleanup pass.
|
|
1601
|
+
- No payload above 2 MB uses a Nango Action; no large body enters MCP JSON.
|
|
1602
|
+
|
|
1603
|
+
### Gate 5 — scheduled automatic ingestion
|
|
1604
|
+
|
|
1605
|
+
- Drive deterministic `connection_sync` adapter enabled.
|
|
1606
|
+
- UI exposes Drive schedule configuration.
|
|
1607
|
+
- Initial and incremental runs produce receipts.
|
|
1608
|
+
- One occurrence/one `scheduledEventUnit` invariant is proven; Google quota units are separately reconciled.
|
|
1609
|
+
|
|
1610
|
+
### Gate 6 — writes
|
|
1611
|
+
|
|
1612
|
+
- The reviewed `upload-document` action ships first; additional mutations require their own exact source/schema/policy entries.
|
|
1613
|
+
- Required scopes verified and reauthorization path proven.
|
|
1614
|
+
- Account action switch, exact grants, audit, and idempotency pass.
|
|
1615
|
+
- Destructive scheduled actions remain blocked until their own approved release.
|
|
1616
|
+
|
|
1617
|
+
### Gate 7 — full surface release
|
|
1618
|
+
|
|
1619
|
+
- Main, scheduler, web, and SDK repos committed/pushed.
|
|
1620
|
+
- Vercel main/scheduler/web production deployments ready.
|
|
1621
|
+
- npm packages published in the required order.
|
|
1622
|
+
- generated MCP manifest/MCPB/README/docs versions agree.
|
|
1623
|
+
- hosted and npm `tools/list` parity checks pass.
|
|
1624
|
+
- integrations and scheduler pages verified on desktop/mobile production.
|
|
1625
|
+
- final release receipt includes every deployment/version/proof result.
|
|
1626
|
+
|
|
1627
|
+
## Done contract
|
|
1628
|
+
|
|
1629
|
+
The work is not “done” because Google Drive shows a green OAuth dot. It is done only when all statements below are simultaneously true:
|
|
1630
|
+
|
|
1631
|
+
- the connection is active with the intended access profile and granted scopes;
|
|
1632
|
+
- live provider functions exist and exactly match reviewed policy;
|
|
1633
|
+
- the user can discover and call safe interactive tools;
|
|
1634
|
+
- action tools are separately scoped, disabled by default, and exactly gated;
|
|
1635
|
+
- “last seven days” and folder/Drive exports are one server-owned operation, not N model calls;
|
|
1636
|
+
- initial inventory and Drive Changes reconciliation cover My Drive and shared drives;
|
|
1637
|
+
- large content streams into private tenant-owned artifacts with explicit per-file outcomes;
|
|
1638
|
+
- scheduled Drive sync resumes, stores changed records/content references, and bills once per occurrence;
|
|
1639
|
+
- connection-native `tools/list` returns sanitized live provider schemas;
|
|
1640
|
+
- root compatibility describe works for Nango and remote MCP transports;
|
|
1641
|
+
- UI, SDK, docs, generated manifests, npm, and Vercel production all reflect the same capability state;
|
|
1642
|
+
- production receipts prove the behavior with real Drive data without exposing secrets or content.
|
|
1643
|
+
|
|
1644
|
+
## Explicit non-goals for the first release
|
|
1645
|
+
|
|
1646
|
+
- Treating Nango's Picker-oriented `documents` sync as proof of complete whole-Drive coverage.
|
|
1647
|
+
- Returning binary/base64 content through MCP or Nango Actions.
|
|
1648
|
+
- Using Google's Developer Preview remote MCP as a backup/export API.
|
|
1649
|
+
- Autonomous Nango polling outside MCP Scraper scheduled-event billing.
|
|
1650
|
+
- Giant ZIP creation inside one Vercel request.
|
|
1651
|
+
- Enabling empty-trash, ownership transfer, public-link mutation, or destructive shared-drive actions in schedules.
|
|
1652
|
+
- Registering every provider action globally on the fixed root MCP.
|
|
1653
|
+
|
|
1654
|
+
## Official references
|
|
1655
|
+
|
|
1656
|
+
- [MCP stable Tool discovery and calls](https://modelcontextprotocol.io/specification/2025-11-25/server/tools)
|
|
1657
|
+
- [MCP stable schema reference](https://modelcontextprotocol.io/specification/2025-11-25/schema)
|
|
1658
|
+
- [MCP cursor pagination](https://modelcontextprotocol.io/specification/2025-11-25/server/utilities/pagination)
|
|
1659
|
+
- [MCP experimental Tasks](https://modelcontextprotocol.io/specification/2025-11-25/basic/utilities/tasks)
|
|
1660
|
+
- [Google Drive OAuth scopes and classifications](https://developers.google.com/workspace/drive/api/guides/api-specific-auth)
|
|
1661
|
+
- [Google Drive files.list](https://developers.google.com/workspace/drive/api/reference/rest/v3/files/list)
|
|
1662
|
+
- [Google Drive change start tokens](https://developers.google.com/workspace/drive/api/reference/rest/v3/changes/getStartPageToken)
|
|
1663
|
+
- [Google Drive changes.list](https://developers.google.com/workspace/drive/api/reference/rest/v3/changes/list)
|
|
1664
|
+
- [Google Drive change-log guidance](https://developers.google.com/workspace/drive/api/guides/about-changes)
|
|
1665
|
+
- [Google Drive download and export guide](https://developers.google.com/workspace/drive/api/guides/manage-downloads)
|
|
1666
|
+
- [Google Drive long-running downloads](https://developers.google.com/workspace/drive/api/guides/long-running-operations)
|
|
1667
|
+
- [Google Drive files.export and 10 MB limit](https://developers.google.com/workspace/drive/api/reference/rest/v3/files/export)
|
|
1668
|
+
- [Google Workspace export MIME mappings](https://developers.google.com/workspace/drive/api/guides/ref-export-formats)
|
|
1669
|
+
- [Google Drive error and backoff guidance](https://developers.google.com/workspace/drive/api/guides/handle-errors)
|
|
1670
|
+
- [Google Drive API usage limits](https://developers.google.com/workspace/drive/api/guides/limits)
|
|
1671
|
+
- [Google official remote Drive MCP setup](https://developers.google.com/workspace/drive/api/guides/configure-mcp-server)
|
|
1672
|
+
- [Google Drive MCP Tool reference](https://developers.google.com/workspace/drive/api/reference/mcp)
|
|
1673
|
+
- [Google Drive MCP file eligibility](https://developers.google.com/workspace/drive/api/guides/drive-mcp-server-file-eligibility)
|
|
1674
|
+
- [Google Drive MCP download_file_content](https://developers.google.com/workspace/drive/api/reference/mcp/tools_list/download_file_content)
|
|
1675
|
+
- [Nango Google Drive integration](https://nango.dev/docs/api-integrations/google-drive)
|
|
1676
|
+
- [Nango Google Drive files sync guide](https://nango.dev/docs/api-integrations/google-drive/how-to-use-google-drive-files-sync)
|
|
1677
|
+
- [Nango Proxy](https://nango.dev/docs/guides/platform/proxy-requests)
|
|
1678
|
+
- [Nango limits, including 2 MB Action output](https://nango.dev/docs/guides/platform/limits)
|
|
1679
|
+
- [Vercel Blob](https://vercel.com/docs/vercel-blob)
|
|
1680
|
+
- [Vercel Blob SDK and multipart uploads](https://vercel.com/docs/vercel-blob/using-blob-sdk)
|