@jskit-ai/connectors-catalog 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +192 -0
- package/docs/ai-upstream-notices.md +51 -0
- package/docs/ai.md +191 -0
- package/docs/airtable.md +130 -0
- package/docs/algolia.md +190 -0
- package/docs/amazon-redshift.md +229 -0
- package/docs/amplitude.md +210 -0
- package/docs/apify.md +162 -0
- package/docs/apollo-io.md +200 -0
- package/docs/asana.md +126 -0
- package/docs/ashby.md +188 -0
- package/docs/atlassian.md +224 -0
- package/docs/attention.md +162 -0
- package/docs/aws-athena.md +150 -0
- package/docs/aws-credentials.md +75 -0
- package/docs/aws-s3.md +193 -0
- package/docs/bigquery.md +146 -0
- package/docs/brevo.md +122 -0
- package/docs/calendly.md +158 -0
- package/docs/canva.md +191 -0
- package/docs/chargebee.md +192 -0
- package/docs/clay.md +194 -0
- package/docs/clickhouse.md +242 -0
- package/docs/confidence-exp.md +74 -0
- package/docs/confidence-flags.md +75 -0
- package/docs/confidence-oauth.md +137 -0
- package/docs/contentful.md +123 -0
- package/docs/databricks.md +180 -0
- package/docs/dbt-semantic-layer.md +175 -0
- package/docs/elevenlabs.md +102 -0
- package/docs/figma.md +132 -0
- package/docs/firebase-cloud-messaging.md +262 -0
- package/docs/firecrawl.md +137 -0
- package/docs/fireflies.md +95 -0
- package/docs/fireworks-ai.md +53 -0
- package/docs/gatewayapi.md +113 -0
- package/docs/gemini-enterprise.md +224 -0
- package/docs/github-api.md +133 -0
- package/docs/gitlab-api.md +138 -0
- package/docs/gmail.md +133 -0
- package/docs/gong.md +205 -0
- package/docs/google-ads.md +156 -0
- package/docs/google-analytics.md +202 -0
- package/docs/google-docs.md +105 -0
- package/docs/google-drive.md +108 -0
- package/docs/google-maps-platform.md +208 -0
- package/docs/google-oauth.md +123 -0
- package/docs/google-search-console.md +124 -0
- package/docs/google-sheets.md +94 -0
- package/docs/google-slides.md +95 -0
- package/docs/granola.md +178 -0
- package/docs/hex.md +161 -0
- package/docs/heygen.md +124 -0
- package/docs/hubspot.md +142 -0
- package/docs/incident-io.md +118 -0
- package/docs/inngest.md +178 -0
- package/docs/klipy.md +149 -0
- package/docs/lexware.md +144 -0
- package/docs/lightspeed.md +193 -0
- package/docs/linear.md +137 -0
- package/docs/linkedin.md +196 -0
- package/docs/logo-dev.md +152 -0
- package/docs/mailgun.md +153 -0
- package/docs/mapbox.md +174 -0
- package/docs/microsoft-excel.md +97 -0
- package/docs/microsoft-fabric.md +249 -0
- package/docs/microsoft-oauth.md +143 -0
- package/docs/microsoft-onedrive.md +76 -0
- package/docs/microsoft-onenote.md +75 -0
- package/docs/microsoft-outlook.md +90 -0
- package/docs/microsoft-powerpoint.md +119 -0
- package/docs/microsoft-sharepoint.md +75 -0
- package/docs/microsoft-teams.md +72 -0
- package/docs/microsoft-word.md +119 -0
- package/docs/miro.md +117 -0
- package/docs/n8n.md +169 -0
- package/docs/notion.md +137 -0
- package/docs/oura.md +161 -0
- package/docs/paddle.md +168 -0
- package/docs/perplexity.md +85 -0
- package/docs/pipedrive.md +120 -0
- package/docs/polar.md +63 -0
- package/docs/posthog.md +227 -0
- package/docs/prestashop.md +151 -0
- package/docs/replicate.md +60 -0
- package/docs/resend.md +153 -0
- package/docs/salesforce.md +234 -0
- package/docs/sanity.md +122 -0
- package/docs/semrush.md +276 -0
- package/docs/sentry.md +112 -0
- package/docs/sevdesk.md +107 -0
- package/docs/shopify.md +206 -0
- package/docs/slack.md +257 -0
- package/docs/snowflake.md +271 -0
- package/docs/storyblok.md +130 -0
- package/docs/stripe.md +77 -0
- package/docs/supabase.md +51 -0
- package/docs/tally.md +105 -0
- package/docs/telegram.md +165 -0
- package/docs/tiktok.md +201 -0
- package/docs/twilio.md +284 -0
- package/docs/twitch.md +428 -0
- package/docs/wave.md +440 -0
- package/docs/wix.md +1249 -0
- package/docs/wiz.md +177 -0
- package/docs/woocommerce.md +400 -0
- package/docs/wordpress-com.md +358 -0
- package/docs/wordpress-self-hosted.md +252 -0
- package/docs/workday.md +292 -0
- package/docs/x-twitter.md +188 -0
- package/docs/xero.md +429 -0
- package/docs/zoho-books.md +214 -0
- package/docs/zoho-crm.md +192 -0
- package/package.json +153 -0
- package/patterns/ai-connections/PATTERN.md +84 -0
- package/patterns/ai-connections/example/ai-model.js +11 -0
- package/patterns/ai-connections/example/perplexity-answer.js +29 -0
- package/patterns/api-key-connection/PATTERN.md +368 -0
- package/patterns/api-key-connection/example/connections.js +18 -0
- package/patterns/api-key-connection/example/integrations.json +19 -0
- package/patterns/assistant-mcp/PATTERN.md +110 -0
- package/patterns/assistant-mcp/example/integrations.json +13 -0
- package/patterns/assistant-mcp-oauth/PATTERN.md +235 -0
- package/patterns/assistant-mcp-oauth/example/integrations.json +102 -0
- package/patterns/aws-storage-queries/PATTERN.md +169 -0
- package/patterns/aws-storage-queries/example/formats/data-formats.js +35 -0
- package/patterns/aws-storage-queries/example/formats/package-lock.json +49 -0
- package/patterns/aws-storage-queries/example/formats/package.json +12 -0
- package/patterns/aws-storage-queries/example/formats/verify-formats.mjs +37 -0
- package/patterns/aws-storage-queries/example/integrations.json +38 -0
- package/patterns/aws-storage-queries/example/s3-transfer.js +34 -0
- package/patterns/event-delivery/PATTERN.md +151 -0
- package/patterns/event-delivery/example/integrations.json +17 -0
- package/patterns/firebase-messaging/PATTERN.md +156 -0
- package/patterns/firebase-messaging/example/server/notifications.js +23 -0
- package/patterns/google-ads-search/PATTERN.md +92 -0
- package/patterns/google-ads-search/example/ads-setup.js +25 -0
- package/patterns/oauth-connection/PATTERN.md +724 -0
- package/patterns/oauth-connection/example/integrations.json +20 -0
- package/patterns/paddle-catalogue/PATTERN.md +78 -0
- package/patterns/paddle-catalogue/example/create-products.js +38 -0
- package/patterns/public-image/PATTERN.md +84 -0
- package/patterns/public-image/example/integrations.json +13 -0
- package/patterns/public-image/example/logo-url.js +16 -0
- package/patterns/redshift-queries/PATTERN.md +150 -0
- package/patterns/redshift-queries/example/integrations.json +33 -0
- package/patterns/source-scanning/PATTERN.md +83 -0
- package/patterns/source-scanning/example/source-scanner.js +13 -0
- package/src/client/firebase-cloud-messaging.js +10 -0
- package/src/client/google-analytics.js +72 -0
- package/src/client/google-maps-platform.js +39 -0
- package/src/client/logo-dev.js +36 -0
- package/src/server/ai.js +46 -0
- package/src/server/airtable.js +67 -0
- package/src/server/algolia.js +63 -0
- package/src/server/amazon-redshift.js +96 -0
- package/src/server/amplitude.js +31 -0
- package/src/server/apify.js +68 -0
- package/src/server/apollo-io.js +85 -0
- package/src/server/asana.js +64 -0
- package/src/server/ashby.js +61 -0
- package/src/server/atlassian.js +32 -0
- package/src/server/attention.js +93 -0
- package/src/server/aws-athena.js +82 -0
- package/src/server/aws-s3.js +49 -0
- package/src/server/awsTransport.js +75 -0
- package/src/server/bigquery.js +97 -0
- package/src/server/brevo.js +97 -0
- package/src/server/calendly.js +77 -0
- package/src/server/canva.js +13 -0
- package/src/server/chargebee.js +88 -0
- package/src/server/clay.js +78 -0
- package/src/server/clickhouse.js +66 -0
- package/src/server/confidence.js +29 -0
- package/src/server/contentful.js +40 -0
- package/src/server/databricks.js +55 -0
- package/src/server/dbt-semantic-layer.js +95 -0
- package/src/server/elevenlabs.js +81 -0
- package/src/server/figma.js +24 -0
- package/src/server/firebase-cloud-messaging.js +111 -0
- package/src/server/firecrawl.js +70 -0
- package/src/server/fireflies.js +31 -0
- package/src/server/fireworks-ai.js +16 -0
- package/src/server/gatewayapi.js +48 -0
- package/src/server/gemini-enterprise.js +47 -0
- package/src/server/github-api.js +90 -0
- package/src/server/gitlab-api.js +67 -0
- package/src/server/gmail.js +60 -0
- package/src/server/gong.js +52 -0
- package/src/server/google-ads-search.js +134 -0
- package/src/server/google-ads.js +80 -0
- package/src/server/google-docs.js +36 -0
- package/src/server/google-drive.js +71 -0
- package/src/server/google-maps-platform.js +90 -0
- package/src/server/google-search-console.js +31 -0
- package/src/server/google-sheets.js +60 -0
- package/src/server/google-slides.js +36 -0
- package/src/server/google.js +61 -0
- package/src/server/granola.js +115 -0
- package/src/server/graphqlOperation.js +15 -0
- package/src/server/hex.js +29 -0
- package/src/server/heygen.js +54 -0
- package/src/server/hubspot.js +85 -0
- package/src/server/incident-io.js +68 -0
- package/src/server/inngest.js +86 -0
- package/src/server/jsonOperation.js +34 -0
- package/src/server/klipy.js +34 -0
- package/src/server/lexware.js +59 -0
- package/src/server/lightspeed.js +82 -0
- package/src/server/linear.js +52 -0
- package/src/server/linkedin.js +74 -0
- package/src/server/mailgun.js +76 -0
- package/src/server/mapbox.js +46 -0
- package/src/server/mcpRegistration.js +41 -0
- package/src/server/mcpTransport.js +60 -0
- package/src/server/microsoft-excel.js +67 -0
- package/src/server/microsoft-fabric.js +81 -0
- package/src/server/microsoft-onedrive.js +58 -0
- package/src/server/microsoft-onenote.js +52 -0
- package/src/server/microsoft-outlook.js +58 -0
- package/src/server/microsoft-powerpoint.js +5 -0
- package/src/server/microsoft-sharepoint.js +46 -0
- package/src/server/microsoft-teams.js +29 -0
- package/src/server/microsoft-word.js +5 -0
- package/src/server/microsoft.js +88 -0
- package/src/server/miro.js +29 -0
- package/src/server/n8n.js +78 -0
- package/src/server/notion.js +76 -0
- package/src/server/oura.js +48 -0
- package/src/server/paddle.js +83 -0
- package/src/server/perplexity.js +15 -0
- package/src/server/pipedrive.js +95 -0
- package/src/server/polar.js +20 -0
- package/src/server/posthog.js +44 -0
- package/src/server/prestashop.js +42 -0
- package/src/server/replicate.js +42 -0
- package/src/server/resend.js +93 -0
- package/src/server/salesforce.js +84 -0
- package/src/server/sanity.js +31 -0
- package/src/server/semrush.js +359 -0
- package/src/server/sentry.js +30 -0
- package/src/server/sevdesk.js +20 -0
- package/src/server/shopify.js +99 -0
- package/src/server/slack.js +143 -0
- package/src/server/snowflake.js +139 -0
- package/src/server/storyblok.js +49 -0
- package/src/server/stripe.js +13 -0
- package/src/server/supabase.js +13 -0
- package/src/server/tally.js +78 -0
- package/src/server/telegram.js +55 -0
- package/src/server/tiktok.js +86 -0
- package/src/server/twilio.js +90 -0
- package/src/server/twitch.js +409 -0
- package/src/server/wave.js +520 -0
- package/src/server/wix.js +1048 -0
- package/src/server/wiz.js +117 -0
- package/src/server/woocommerce.js +412 -0
- package/src/server/wordpress-com.js +236 -0
- package/src/server/wordpress-self-hosted.js +134 -0
- package/src/server/workday.js +136 -0
- package/src/server/x-twitter.js +110 -0
- package/src/server/xero.js +397 -0
- package/src/server/zoho-books.js +90 -0
- package/src/server/zoho-crm.js +89 -0
- package/src/server/zohoOAuth.js +21 -0
- package/src/shared/ai-models.json +8049 -0
- package/src/shared/ai.js +63 -0
- package/src/shared/algolia.js +29 -0
- package/src/shared/amazon-redshift.js +51 -0
- package/src/shared/amplitude.js +28 -0
- package/src/shared/atlassian.js +43 -0
- package/src/shared/aws.js +72 -0
- package/src/shared/canva.js +66 -0
- package/src/shared/chargebee.js +26 -0
- package/src/shared/clickhouse.js +38 -0
- package/src/shared/confidence.js +28 -0
- package/src/shared/databricks.js +33 -0
- package/src/shared/dbt-semantic-layer.js +32 -0
- package/src/shared/definitions.js +140 -0
- package/src/shared/figma.js +21 -0
- package/src/shared/firebase-cloud-messaging.js +45 -0
- package/src/shared/gemini-enterprise.js +33 -0
- package/src/shared/gong.js +33 -0
- package/src/shared/google-ads-search.js +31 -0
- package/src/shared/google-ads.js +28 -0
- package/src/shared/google-maps-platform.js +28 -0
- package/src/shared/google.js +133 -0
- package/src/shared/granola.js +25 -0
- package/src/shared/hex.js +28 -0
- package/src/shared/inngest.js +35 -0
- package/src/shared/lightspeed.js +75 -0
- package/src/shared/linkedin.js +27 -0
- package/src/shared/logo-dev.js +21 -0
- package/src/shared/mapbox.js +38 -0
- package/src/shared/mcp.js +92 -0
- package/src/shared/microsoft-fabric.js +47 -0
- package/src/shared/microsoft.js +231 -0
- package/src/shared/miro.js +27 -0
- package/src/shared/oura.js +23 -0
- package/src/shared/posthog.js +33 -0
- package/src/shared/prestashop.js +23 -0
- package/src/shared/salesforce.js +36 -0
- package/src/shared/semrush.js +24 -0
- package/src/shared/shopify.js +51 -0
- package/src/shared/siteUrl.js +12 -0
- package/src/shared/slack.js +88 -0
- package/src/shared/snowflake.js +55 -0
- package/src/shared/tiktok.js +22 -0
- package/src/shared/tokens.js +747 -0
- package/src/shared/twilio.js +41 -0
- package/src/shared/twitch.js +49 -0
- package/src/shared/wave.js +52 -0
- package/src/shared/wix.js +40 -0
- package/src/shared/wiz.js +41 -0
- package/src/shared/wordpress-com.js +30 -0
- package/src/shared/wordpress.js +69 -0
- package/src/shared/workday.js +50 -0
- package/src/shared/x-twitter.js +19 -0
- package/src/shared/xero.js +46 -0
- package/src/shared/zoho-books.js +39 -0
- package/src/shared/zoho-crm.js +43 -0
- package/src/shared/zoho-regions.js +12 -0
- package/test/ai.test.js +136 -0
- package/test/airtable.test.js +125 -0
- package/test/algolia.test.js +207 -0
- package/test/amazon-redshift.test.js +272 -0
- package/test/amplitude.test.js +239 -0
- package/test/apify.test.js +93 -0
- package/test/apollo-io.test.js +125 -0
- package/test/asana.test.js +99 -0
- package/test/ashby.test.js +97 -0
- package/test/atlassian.test.js +249 -0
- package/test/attention.test.js +79 -0
- package/test/aws.test.js +405 -0
- package/test/brevo.test.js +71 -0
- package/test/calendly.test.js +136 -0
- package/test/canva.test.js +258 -0
- package/test/chargebee.test.js +195 -0
- package/test/clickhouse.test.js +278 -0
- package/test/confidence.test.js +205 -0
- package/test/databricks.test.js +253 -0
- package/test/dbt-semantic-layer.test.js +221 -0
- package/test/designMcp.test.js +336 -0
- package/test/elevenlabs.test.js +52 -0
- package/test/firebase-cloud-messaging.test.js +242 -0
- package/test/firecrawl.test.js +66 -0
- package/test/gemini-enterprise.test.js +218 -0
- package/test/github-api.test.js +167 -0
- package/test/gitlab-api.test.js +160 -0
- package/test/gong.test.js +192 -0
- package/test/google-ads.test.js +350 -0
- package/test/google-maps-platform.test.js +191 -0
- package/test/google.test.js +710 -0
- package/test/granola.test.js +240 -0
- package/test/hex.test.js +253 -0
- package/test/heygen.test.js +121 -0
- package/test/hubspot.test.js +103 -0
- package/test/inngest.test.js +219 -0
- package/test/lightspeed.test.js +250 -0
- package/test/linear.test.js +146 -0
- package/test/linkedin.test.js +243 -0
- package/test/logo-dev.test.js +79 -0
- package/test/mailgun.test.js +107 -0
- package/test/mapbox.test.js +181 -0
- package/test/mcp.test.js +227 -0
- package/test/microsoft-fabric.test.js +258 -0
- package/test/microsoft.test.js +554 -0
- package/test/n8nOAuth.test.js +188 -0
- package/test/notionOAuth.test.js +183 -0
- package/test/oura.test.js +211 -0
- package/test/paddleCatalogue.test.js +51 -0
- package/test/paddlePermissions.test.js +101 -0
- package/test/perplexityComposition.test.js +39 -0
- package/test/pipedriveOAuth.test.js +140 -0
- package/test/posthog.test.js +200 -0
- package/test/prestashop.test.js +207 -0
- package/test/resend.test.js +163 -0
- package/test/salesforce.test.js +276 -0
- package/test/semrush.test.js +479 -0
- package/test/sentry.test.js +35 -0
- package/test/shopify.test.js +247 -0
- package/test/slack.test.js +320 -0
- package/test/snowflake.test.js +391 -0
- package/test/tally.test.js +75 -0
- package/test/telegram.test.js +58 -0
- package/test/tiktok.test.js +243 -0
- package/test/tokens.test.js +1079 -0
- package/test/twilio.test.js +221 -0
- package/test/twitch.test.js +573 -0
- package/test/wave.test.js +678 -0
- package/test/wix.test.js +1099 -0
- package/test/wiz.test.js +185 -0
- package/test/wordpress-com.test.js +469 -0
- package/test/wordpress.test.js +660 -0
- package/test/workday.test.js +314 -0
- package/test/x-twitter.test.js +256 -0
- package/test/xero.test.js +700 -0
- package/test/zoho-books.test.js +253 -0
- package/test/zoho-crm.test.js +223 -0
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
# AWS Athena
|
|
2
|
+
|
|
3
|
+
Import `awsAthenaProvider` from
|
|
4
|
+
`@jskit-ai/connectors-catalog/server/aws-athena`. Use the same connection service,
|
|
5
|
+
Feature, file storage and shared configuration form as other API-key providers.
|
|
6
|
+
The fragment executes Athena SQL operations through the official AWS SDK.
|
|
7
|
+
|
|
8
|
+
## Portable configuration
|
|
9
|
+
|
|
10
|
+
```json
|
|
11
|
+
{
|
|
12
|
+
"schemaVersion": 1,
|
|
13
|
+
"registrations": {},
|
|
14
|
+
"integrations": {
|
|
15
|
+
"queries": {
|
|
16
|
+
"provider": "aws-athena",
|
|
17
|
+
"accountMode": "shared",
|
|
18
|
+
"scopes": [],
|
|
19
|
+
"authentication": {
|
|
20
|
+
"method": "api-key",
|
|
21
|
+
"secretRef": "env:AWS_SECRET_ACCESS_KEY"
|
|
22
|
+
},
|
|
23
|
+
"settings": {
|
|
24
|
+
"region": "ap-southeast-2",
|
|
25
|
+
"accessKeyIdRef": "env:AWS_ACCESS_KEY_ID",
|
|
26
|
+
"workgroup": "reports",
|
|
27
|
+
"resultLocation": "s3://query-results/reports/"
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
`workgroup` defaults to `primary`. Omit `resultLocation` when the workgroup
|
|
35
|
+
supplies result storage. Enforced workgroup configuration takes precedence over
|
|
36
|
+
client result settings, as documented in
|
|
37
|
+
[Override client-side settings](https://docs.aws.amazon.com/athena/latest/ug/workgroups-settings-override.html).
|
|
38
|
+
Temporary credentials also need `settings.sessionTokenRef`. `assistant`
|
|
39
|
+
ownership is supported; application users still share the configured AWS identity.
|
|
40
|
+
|
|
41
|
+
## Administrator setup
|
|
42
|
+
|
|
43
|
+
1. In the Athena console, select the intended AWS account and region.
|
|
44
|
+
2. Open **Workgroups** in the navigation pane. Select an existing SQL workgroup or
|
|
45
|
+
choose **Create workgroup**, enter its name, and select the Athena SQL engine and **AWS Identity and Access Management (IAM)** authentication.
|
|
46
|
+
3. Configure result storage for that workgroup. For S3-backed results, select a
|
|
47
|
+
bucket/prefix with the intended encryption and ownership. Set a per-query
|
|
48
|
+
data scan limit; choose **Override client-side settings** if the workgroup
|
|
49
|
+
must enforce those settings. Choose **Create workgroup** or save your edits.
|
|
50
|
+
4. Give the backend identity `athena:GetWorkGroup`, `StartQueryExecution`,
|
|
51
|
+
`GetQueryExecution`, `GetQueryResults` and `StopQueryExecution` on the intended
|
|
52
|
+
workgroup. Grant only the required source data, catalog, result storage and
|
|
53
|
+
encryption access. Permissions depend on the actual query's data sources.
|
|
54
|
+
5. Follow [AWS credential setup](aws-credentials.md), enter the references,
|
|
55
|
+
region and workgroup in Vibe64 or the JSON file, and save.
|
|
56
|
+
6. In the editor, use **Set credential in Env** for each saved reference and enter
|
|
57
|
+
its value, then return and choose **Connect account**. CLI callers resolve the
|
|
58
|
+
same references from their private environment and call `connectApiKey`.
|
|
59
|
+
Temporary credentials require all three values to be renewed together.
|
|
60
|
+
The connection check reads workgroup metadata; it does
|
|
61
|
+
not run a query, verify table access or prove result-bucket permissions.
|
|
62
|
+
|
|
63
|
+
Console configuration follows [Create a workgroup](https://docs.aws.amazon.com/athena/latest/ug/creating-workgroups.html).
|
|
64
|
+
An authorized AI can automate workgroup creation with
|
|
65
|
+
[CreateWorkGroup](https://docs.aws.amazon.com/athena/latest/APIReference/API_CreateWorkGroup.html)
|
|
66
|
+
and separately configure IAM, Glue/S3 resources and encryption. It cannot grant
|
|
67
|
+
itself account access. Neither provisioning nor credential renewal is part of
|
|
68
|
+
this fragment.
|
|
69
|
+
|
|
70
|
+
For Online and public operation there is **no OAuth callback URL** in this
|
|
71
|
+
mode. See [AWS ownership and quota isolation](aws-credentials.md#online-public-editor-and-independent-cli).
|
|
72
|
+
Separate workgroups do not isolate account service quotas. The application
|
|
73
|
+
operator supplies AWS credentials, chooses resources and controls their budget.
|
|
74
|
+
The same application-owned setup works from CLI or editor without a Vibe64
|
|
75
|
+
managed service.
|
|
76
|
+
|
|
77
|
+
## Runtime contract
|
|
78
|
+
|
|
79
|
+
| Operation | Inputs | Result |
|
|
80
|
+
|---|---|---|
|
|
81
|
+
| `workgroup.get` | None | Configured workgroup metadata, including enabled/disabled state |
|
|
82
|
+
| `catalogs.list` | Optional `maxResults` 2–50 (default 50), opaque `nextToken` | Catalog summaries |
|
|
83
|
+
| `databases.list` | `catalog`; optional `maxResults` 1–50, `nextToken` | Database names and metadata |
|
|
84
|
+
| `tables.list` | `catalog`, `database`; optional `maxResults` 1–50, `nextToken` | Table metadata, columns and partition-key types |
|
|
85
|
+
| `query.start` | `sql`, `clientRequestToken` (32–128 chars); optional `database`, `catalog` | `QueryExecutionId` |
|
|
86
|
+
| `query.status` | `queryId` | Query execution, status, statistics and result configuration |
|
|
87
|
+
| `query.results` | `queryId`; optional `maxResults` 1–1000 (default 100), opaque `nextToken` | One DATA_ROWS result page |
|
|
88
|
+
| `query.cancel` | `queryId` | Empty success receipt after StopQueryExecution |
|
|
89
|
+
|
|
90
|
+
Query statements are not restricted to SELECT. The host's authorization policy
|
|
91
|
+
must approve the exact SQL and bound data access before invocation. SQL is
|
|
92
|
+
preserved, limited to 262144 UTF-8 bytes and never submitted during connection.
|
|
93
|
+
A client request token must identify one application-authorized logical query;
|
|
94
|
+
reuse the same token and input when resolving an uncertain submission. The SDK
|
|
95
|
+
is configured for one attempt; there is no automatic query replay. AWS describes
|
|
96
|
+
token reuse in [StartQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html).
|
|
97
|
+
|
|
98
|
+
Status, results and cancellation first fetch the execution and reject a query
|
|
99
|
+
outside the configured workgroup. This does not isolate users within a shared
|
|
100
|
+
workgroup. The host must maintain query ownership and authorize each `queryId`.
|
|
101
|
+
The fragment returns queued, running, succeeded, failed and cancelled states;
|
|
102
|
+
it does not poll, schedule or invent completion. See
|
|
103
|
+
[GetQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html).
|
|
104
|
+
|
|
105
|
+
Results preserve column metadata, header rows, empty cells and string values.
|
|
106
|
+
Do not coerce large numbers or silently drop the first row. Follow `NextToken`
|
|
107
|
+
explicitly; one page is not the full result. S3-backed results require S3 access
|
|
108
|
+
as well as Athena permissions. See
|
|
109
|
+
[GetQueryResults](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html).
|
|
110
|
+
|
|
111
|
+
Cancelling the local HTTP call or disconnecting does not stop an AWS query.
|
|
112
|
+
`query.cancel` explicitly requests
|
|
113
|
+
[StopQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_StopQueryExecution.html);
|
|
114
|
+
read status afterwards if final cancellation must be confirmed. Catalog
|
|
115
|
+
provisioning, prepared statements, execution parameters, Spark, identity-center
|
|
116
|
+
federation and result manifest operations are outside this fragment.
|
|
117
|
+
|
|
118
|
+
Local tests exercise real SDK serialization/signing against controlled HTTP,
|
|
119
|
+
file restart, permission boundaries, job states and cancellation. No live SQL,
|
|
120
|
+
AWS account provisioning or generated application is used.
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
## Catalog browsing
|
|
124
|
+
|
|
125
|
+
Use `catalogs.list`, then `databases.list` for the selected catalog and
|
|
126
|
+
`tables.list` for its selected database. Each operation returns one native AWS
|
|
127
|
+
page; pass `NextToken` unchanged to continue. Column types remain provider
|
|
128
|
+
strings, including decimal precision and nested types. These calls do not run SQL.
|
|
129
|
+
|
|
130
|
+
The configured workgroup is sent on each request, but does not restrict which
|
|
131
|
+
catalogs are visible to that IAM identity. Authorize the exact catalog/database
|
|
132
|
+
and returned metadata in the application, separately from query ownership.
|
|
133
|
+
Grant `athena:ListDataCatalogs`, `athena:ListDatabases` and
|
|
134
|
+
`athena:ListTableMetadata` as needed. Glue-backed catalogs also require the
|
|
135
|
+
appropriate `glue:GetDatabase`, `glue:GetDatabases`, `glue:GetTable` and
|
|
136
|
+
`glue:GetTables` resource grants. Do not broaden them automatically after denial.
|
|
137
|
+
|
|
138
|
+
```js
|
|
139
|
+
const databases = await service.invoke({ context, integrationId: "queries",
|
|
140
|
+
operation: "databases.list", input: { catalog: authorizedCatalog } });
|
|
141
|
+
const tables = await service.invoke({ context, integrationId: "queries",
|
|
142
|
+
operation: "tables.list",
|
|
143
|
+
input: { catalog: authorizedCatalog, database: authorizedDatabase } });
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
Native implementations in other frameworks use AWS
|
|
147
|
+
[ListDataCatalogs](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDataCatalogs.html),
|
|
148
|
+
[ListDatabases](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDatabases.html)
|
|
149
|
+
and [ListTableMetadata](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListTableMetadata.html)
|
|
150
|
+
with the same project credential bindings and application authorization.
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# AWS credentials and ownership
|
|
2
|
+
|
|
3
|
+
S3 and Athena use AWS Signature Version 4, implemented by the official AWS SDK.
|
|
4
|
+
The shared runtime resolves explicit credential references for each operation.
|
|
5
|
+
It never falls back to environment names, AWS profiles, instance metadata or the
|
|
6
|
+
machine operator's account. CLI and Vibe64 use the same configuration and file
|
|
7
|
+
connection store. This is useful library code; provisioning remains host work.
|
|
8
|
+
|
|
9
|
+
## Create or obtain credentials
|
|
10
|
+
|
|
11
|
+
Prefer temporary role credentials. An administrator supplies a role with the
|
|
12
|
+
resource permissions listed in the provider guide; an authorized host can use
|
|
13
|
+
[STS AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)
|
|
14
|
+
to obtain access key ID, secret access key and session token. Trust policies,
|
|
15
|
+
external IDs for third-party access, MFA requirements and expiration belong to
|
|
16
|
+
that setup. This fragment accepts the resulting three bindings; it does not
|
|
17
|
+
assume roles or refresh STS credentials itself. The resolver must return a
|
|
18
|
+
consistent credential set and renew it before expiration.
|
|
19
|
+
|
|
20
|
+
For a deliberately selected IAM-user key:
|
|
21
|
+
|
|
22
|
+
1. Sign into the AWS console as that IAM user, in the intended account.
|
|
23
|
+
2. Open your account menu at the top right, then **Security credentials**.
|
|
24
|
+
3. Under **Access keys**, choose **Create access key**. Review the alternatives.
|
|
25
|
+
4. If a long-lived key is appropriate, select **Other**, then **Next**.
|
|
26
|
+
5. Add a description identifying this application and environment. Choose
|
|
27
|
+
**Create access key**.
|
|
28
|
+
6. On **Retrieve access keys**, save the ID and secret into the backend's secret
|
|
29
|
+
environment. AWS reveals the secret only at creation. Choose **Done**.
|
|
30
|
+
7. Put references such as `env:AWS_ACCESS_KEY_ID` and
|
|
31
|
+
`env:AWS_SECRET_ACCESS_KEY` in the integration configuration. Leave the
|
|
32
|
+
optional session-token reference absent for long-lived IAM-user keys.
|
|
33
|
+
|
|
34
|
+
These are AWS's [documented console steps](https://docs.aws.amazon.com/IAM/latest/UserGuide/access-key-self-managed.html).
|
|
35
|
+
Use an IAM identity restricted to the intended resources, not an account root key.
|
|
36
|
+
Account enrollment, billing and administrator access are prerequisites; adding
|
|
37
|
+
an integration does not create an AWS account or confer administrator rights.
|
|
38
|
+
|
|
39
|
+
## Online, public editor and independent CLI
|
|
40
|
+
|
|
41
|
+
There is **no OAuth registration, redirect URI or universal callback URL** for
|
|
42
|
+
these credential modes. The app's VPS and custom domain do not change the AWS
|
|
43
|
+
account, region or identity. Browser S3 access can require a CORS update when the
|
|
44
|
+
app changes domain; server-side Athena calls do not use browser CORS.
|
|
45
|
+
|
|
46
|
+
The application owner supplies the AWS identity and resource set. CLI and editor
|
|
47
|
+
configuration use the same application-owned Env bindings. Hosted and installed
|
|
48
|
+
editors do not supply shared AWS credentials. Two access keys or roles in one
|
|
49
|
+
account do not create separate quota pools.
|
|
50
|
+
|
|
51
|
+
After saving the configuration, Vibe64 provides a **Set credential in Env**
|
|
52
|
+
link for the access key ID, secret access key and any configured session token.
|
|
53
|
+
Each link opens that project's Env screen with the referenced variable name.
|
|
54
|
+
Only references are saved in the integration file. Clearing the optional token
|
|
55
|
+
reference removes its shortcut; it does not delete the existing Env value.
|
|
56
|
+
|
|
57
|
+
AWS's [Athena quota documentation](https://docs.aws.amazon.com/athena/latest/ug/service-limits.html)
|
|
58
|
+
explicitly shares service quotas across an account's workgroups. Separate
|
|
59
|
+
workgroups are useful for policies and query limits; they do not isolate account
|
|
60
|
+
capacity. Account and role provisioning are operator responsibilities, separate
|
|
61
|
+
from these runtime adapters.
|
|
62
|
+
|
|
63
|
+
## Can an AI automate setup?
|
|
64
|
+
|
|
65
|
+
With explicitly authorized administrative credentials, an AI can compose and
|
|
66
|
+
apply resource policies, create service resources, and call
|
|
67
|
+
[CreateAccessKey](https://docs.aws.amazon.com/IAM/latest/APIReference/API_CreateAccessKey.html)
|
|
68
|
+
for an existing IAM user. Secret output must be captured directly into the
|
|
69
|
+
chosen secret store. Resource creation and policy changes are separate from
|
|
70
|
+
runtime verification; these adapters never perform them on Connect.
|
|
71
|
+
|
|
72
|
+
An AI cannot bootstrap trust or permissions that the owner has not granted.
|
|
73
|
+
The current package includes guidance and runtime operations, not an automated
|
|
74
|
+
AWS account factory, IAM policy deployment or STS renewal service. Follow the
|
|
75
|
+
[S3](aws-s3.md) or [Athena](aws-athena.md) setup for the actual resources.
|
package/docs/aws-s3.md
ADDED
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
# AWS S3
|
|
2
|
+
|
|
3
|
+
Import `awsS3Provider` from `@jskit-ai/connectors-catalog/server/aws-s3` and use
|
|
4
|
+
`createConnectionService` or the ordinary JSKIT connector Feature. The shared
|
|
5
|
+
catalogue definition drives Vibe64's form. Runtime state can use encrypted JSON
|
|
6
|
+
files; no database or generated application is required.
|
|
7
|
+
|
|
8
|
+
## Portable configuration
|
|
9
|
+
|
|
10
|
+
```json
|
|
11
|
+
{
|
|
12
|
+
"schemaVersion": 1,
|
|
13
|
+
"registrations": {},
|
|
14
|
+
"integrations": {
|
|
15
|
+
"files": {
|
|
16
|
+
"provider": "aws-s3",
|
|
17
|
+
"displayName": "Application files",
|
|
18
|
+
"accountMode": "shared",
|
|
19
|
+
"scopes": ["read"],
|
|
20
|
+
"authentication": {
|
|
21
|
+
"method": "api-key",
|
|
22
|
+
"secretRef": "env:AWS_SECRET_ACCESS_KEY"
|
|
23
|
+
},
|
|
24
|
+
"settings": {
|
|
25
|
+
"region": "ap-southeast-2",
|
|
26
|
+
"bucket": "my-app-files",
|
|
27
|
+
"accessKeyIdRef": "env:AWS_ACCESS_KEY_ID"
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Optional `settings.sessionTokenRef` binds temporary credentials. The form
|
|
35
|
+
initially selects read and write, matching the captured two-permission setup;
|
|
36
|
+
remove write for a reader. Read access is mandatory: the configuration validator
|
|
37
|
+
rejects its absence and the editor keeps that checkbox selected and disabled.
|
|
38
|
+
These are local operation limits, not an IAM grant.
|
|
39
|
+
The returned `grantedScopes` remains empty because AWS did not issue OAuth
|
|
40
|
+
consent. `assistant` ownership is also supported; neither mode represents a
|
|
41
|
+
separate AWS account for each end user.
|
|
42
|
+
|
|
43
|
+
## Administrator setup
|
|
44
|
+
|
|
45
|
+
1. Open **Amazon S3 > General purpose buckets** in the intended AWS account.
|
|
46
|
+
Select an existing bucket, or choose **Create bucket**, select the region,
|
|
47
|
+
supply a valid unique bucket name and create it. Retain blocked public access
|
|
48
|
+
and disabled ACLs unless your application specifically requires otherwise.
|
|
49
|
+
2. Copy the bucket name and its region into the form. This fragment supports the
|
|
50
|
+
18 regions listed by its shared definition, using the commercial AWS
|
|
51
|
+
partition and general purpose buckets in the shared global namespace.
|
|
52
|
+
3. Give the backend identity `s3:ListBucket` on the bucket ARN and
|
|
53
|
+
`s3:GetObject` on the permitted object ARNs. Add `s3:PutObject` only for uploads.
|
|
54
|
+
Encryption or cross-account policies can require additional permissions.
|
|
55
|
+
4. Follow [AWS credential setup](aws-credentials.md). Store references in the
|
|
56
|
+
form, save the file, then let the runtime call `connectApiKey`.
|
|
57
|
+
5. For browser fetches or uploads, open the bucket's **Permissions** tab. Under
|
|
58
|
+
**Cross-origin resource sharing (CORS)** choose **Edit**, add the app's exact
|
|
59
|
+
origins and needed methods/headers, then **Save changes**. Add a verified
|
|
60
|
+
custom-domain origin when it becomes active. CORS does not grant object access.
|
|
61
|
+
|
|
62
|
+
For the native Fetch example, this is a starting CORS document. Replace the
|
|
63
|
+
illustrative origin with the application's real HTTPS origin, without a path
|
|
64
|
+
or trailing slash. Remove PUT for a download-only app. Preserve any existing
|
|
65
|
+
rules required by other applications using the bucket.
|
|
66
|
+
|
|
67
|
+
```json
|
|
68
|
+
[
|
|
69
|
+
{
|
|
70
|
+
"AllowedOrigins": ["https://your-app.example.com"],
|
|
71
|
+
"AllowedMethods": ["GET", "PUT"],
|
|
72
|
+
"AllowedHeaders": ["Content-Type"],
|
|
73
|
+
"ExposeHeaders": ["ETag"],
|
|
74
|
+
"MaxAgeSeconds": 300
|
|
75
|
+
}
|
|
76
|
+
]
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
If your upload code adds checksum or other request headers, list those headers
|
|
80
|
+
too. A server-side transfer does not require browser CORS. Keep the AWS keys on
|
|
81
|
+
the server; the browser receives only the URL for its authorized object.
|
|
82
|
+
|
|
83
|
+
The [bucket API guide](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html)
|
|
84
|
+
explains bucket types, regions and public-access settings. Browser setup follows
|
|
85
|
+
[AWS's CORS instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html).
|
|
86
|
+
An authorized AI can use CreateBucket, IAM policies and
|
|
87
|
+
[PutBucketCors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html)
|
|
88
|
+
to perform these administrative steps. Provisioning is not an adapter operation.
|
|
89
|
+
For application-owned credentials, quota boundaries and the absence of a callback URL,
|
|
90
|
+
see [AWS ownership](aws-credentials.md#online-public-editor-and-independent-cli).
|
|
91
|
+
|
|
92
|
+
## Runtime operations
|
|
93
|
+
|
|
94
|
+
| Operation | Inputs | Behavior |
|
|
95
|
+
|---|---|---|
|
|
96
|
+
| `objects.list` | Optional `maxKeys` 1–1000 (default 100), `prefix`, delimiter `/`, opaque `continuationToken` | One ListObjectsV2 page in the configured bucket; also the connection check |
|
|
97
|
+
| `objects.downloadUrl` | Exact `key`; optional `expiresInSeconds` 1–900 (default 300) | Returns `{url, method: "GET", expiresInSeconds}` |
|
|
98
|
+
| `objects.uploadUrl` | Same fields; requires local `write` permission | Returns `{url, method: "PUT", expiresInSeconds}` |
|
|
99
|
+
|
|
100
|
+
Listing preserves AWS's response shape and uses `EncodingType: "url"`.
|
|
101
|
+
Decode the listed `Contents[].Key` and `CommonPrefixes[].Prefix` with
|
|
102
|
+
`decodeURIComponent` once before displaying or passing a key to a URL operation;
|
|
103
|
+
do not decode opaque continuation tokens. Empty pages are valid; continue using
|
|
104
|
+
`NextContinuationToken` when `IsTruncated` is true. See
|
|
105
|
+
[ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html).
|
|
106
|
+
|
|
107
|
+
Keys preserve spaces, Unicode, plus signs and literal percent signs. Keys over
|
|
108
|
+
1024 UTF-8 bytes, and standalone `.` or `..` path segments, are rejected to avoid
|
|
109
|
+
URL normalization changing object identity. These are deliberate fragment bounds
|
|
110
|
+
based on [AWS key behavior](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html).
|
|
111
|
+
No directory buckets, access-point ARNs, alternate endpoints, multipart uploads,
|
|
112
|
+
version selection, delete, ACL or encryption-header operations are implemented.
|
|
113
|
+
|
|
114
|
+
Presigning is local. It does not prove that an object exists, a transfer
|
|
115
|
+
succeeded or IAM permits it. Use the returned method and unmodified URL without
|
|
116
|
+
an Authorization header. PUT sends the file body and can overwrite that key.
|
|
117
|
+
URLs are bearer credentials, reusable until effective expiration; temporary
|
|
118
|
+
credentials can expire first. Do not log them or put them in public source.
|
|
119
|
+
Local disconnect prevents future use through the service but cannot revoke an
|
|
120
|
+
already issued URL. See [presigned URL behavior](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html).
|
|
121
|
+
|
|
122
|
+
The host must authorize the exact key/prefix before invocation. A shared bucket
|
|
123
|
+
connection alone does not isolate application users. SDK calls use one attempt,
|
|
124
|
+
explicit credentials and a fixed regional origin; wrong-region responses and
|
|
125
|
+
expired credentials remain observable. Local tests use controlled HTTP with
|
|
126
|
+
real SDK signing and encrypted file persistence. They do not access a live bucket
|
|
127
|
+
or transfer files through issued URLs.
|
|
128
|
+
|
|
129
|
+
## Application-owned file transfers
|
|
130
|
+
|
|
131
|
+
The tested [transfer example](../patterns/aws-storage-queries/example/s3-transfer.js)
|
|
132
|
+
composes the existing connection service with native Fetch. Copy it into an
|
|
133
|
+
application server module or CLI; it needs no Vibe64 process. Supply the same
|
|
134
|
+
authorized context and integration ID used by the service, an exact permitted
|
|
135
|
+
key, and a timeout/cancellation signal. The upload body is a Blob, ArrayBuffer
|
|
136
|
+
or typed array. Downloads return a Response whose body can be streamed to a file
|
|
137
|
+
or a format reader. Await body consumption before reporting completion.
|
|
138
|
+
|
|
139
|
+
```js
|
|
140
|
+
const response = await transferS3Object({ service, context,
|
|
141
|
+
integrationId: "files", key: authorizedKey, direction: "download",
|
|
142
|
+
signal: AbortSignal.timeout(60_000) });
|
|
143
|
+
// For a bounded JSON document; validate its schema in your application.
|
|
144
|
+
const document = await response.json();
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
For CSV and Parquet, preserve the response bytes and use the application's
|
|
148
|
+
selected CSV/Parquet reader; S3 does not interpret these formats. The
|
|
149
|
+
[native format example](../patterns/aws-storage-queries/example/formats/data-formats.js)
|
|
150
|
+
implements JSON, CSV and Parquet encoding/decoding through native libraries.
|
|
151
|
+
Its separate package manifest and lockfile are illustrative application
|
|
152
|
+
dependencies, not dependencies of the connector runtime.
|
|
153
|
+
|
|
154
|
+
Copy the `example/formats` directory to an isolated working directory, run
|
|
155
|
+
`npm ci --ignore-scripts`, then `npm test`. This exercises only the format
|
|
156
|
+
libraries; it does not generate or launch an application or contact AWS.
|
|
157
|
+
An existing application can instead install the listed libraries using its
|
|
158
|
+
normal dependency workflow and adapt `data-formats.js` into its own server code.
|
|
159
|
+
|
|
160
|
+
```js
|
|
161
|
+
const encoded = encodeDataFile("csv", [
|
|
162
|
+
["id", "description"], ["001", "A description, with punctuation"]
|
|
163
|
+
]);
|
|
164
|
+
await transferS3Object({ service, context, integrationId: "files",
|
|
165
|
+
key: authorizedKey, direction: "upload", body: encoded,
|
|
166
|
+
signal: AbortSignal.timeout(60_000) });
|
|
167
|
+
const downloaded = await transferS3Object({ service, context,
|
|
168
|
+
integrationId: "files", key: authorizedKey, direction: "download",
|
|
169
|
+
signal: AbortSignal.timeout(60_000) });
|
|
170
|
+
const rows = await decodeDataFile("csv", new Uint8Array(await downloaded.arrayBuffer()));
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
CSV preserves row arrays, quoting, newlines and leading zeroes; the application
|
|
174
|
+
decides which row is the header and validates its business schema. Parquet
|
|
175
|
+
encoding accepts explicit native `{name, type, data}` columns; decoding returns
|
|
176
|
+
row objects. The example uses in-memory parsing with an 8 MiB input limit.
|
|
177
|
+
That limit is checked after download and is not a network-transfer or decompressed
|
|
178
|
+
memory limit. Large or untrusted datasets need the application's bounded stream,
|
|
179
|
+
row/column selection and resource policy. Other Parquet codecs and schemas should
|
|
180
|
+
use the native reader's documented options rather than a new connector API.
|
|
181
|
+
|
|
182
|
+
The example refuses redirects, omits cookies, sends no Authorization header,
|
|
183
|
+
and does not retry transfers. A failed PUT connection can mean an unknown
|
|
184
|
+
outcome: inspect the intended object before deciding to repeat the upload.
|
|
185
|
+
HTTP failures remain failures even when URL signing succeeded. A stream can
|
|
186
|
+
also fail after receiving successful response headers. Follow the provider's
|
|
187
|
+
[upload instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)
|
|
188
|
+
when using a URL in another framework or a browser.
|
|
189
|
+
|
|
190
|
+
Controlled tests exercise real SDK signing followed by fixture GET/PUT
|
|
191
|
+
transfers, binary chunk consumption, transfer errors and denied authorization.
|
|
192
|
+
Separate format checks exercise JSON, quoted CSV and typed nullable Parquet
|
|
193
|
+
content with the real libraries. Neither suite establishes live AWS interoperability.
|
package/docs/bigquery.md
ADDED
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# BigQuery
|
|
2
|
+
|
|
3
|
+
Import `bigqueryProvider` from `@jskit-ai/connectors-catalog/server/bigquery`.
|
|
4
|
+
**Existing-scope delivery: delegated OAuth only.** Federation is not implemented. Query
|
|
5
|
+
submission and results use the configured project. The project-list connection
|
|
6
|
+
check does not establish job or dataset permissions.
|
|
7
|
+
|
|
8
|
+
This fragment lists projects and datasets, reads table metadata, submits GoogleSQL queries and retrieves results
|
|
9
|
+
using delegated Google OAuth. The generated application owns these operations.
|
|
10
|
+
|
|
11
|
+
## Registration and manual setup
|
|
12
|
+
|
|
13
|
+
1. Follow the numbered [Google OAuth setup](google-oauth.md) to choose/create
|
|
14
|
+
the Cloud project and configure branding, audience and test users.
|
|
15
|
+
2. In **APIs & Services → Library**, enable **BigQuery API**
|
|
16
|
+
(`bigquery.googleapis.com`). Create a Web application OAuth client with the
|
|
17
|
+
exact callback URI served by the backend.
|
|
18
|
+
3. In **Data Access**, add
|
|
19
|
+
`https://www.googleapis.com/auth/bigquery`. Store client secret and
|
|
20
|
+
callback URL in Env, then save their references and client ID in the shared
|
|
21
|
+
registration. Choose provider `bigquery` and the OAuth registration reference.
|
|
22
|
+
4. Enter **Google Cloud project ID**: the project intended to run and pay for
|
|
23
|
+
queries. It may differ from the dataset project. This is saved as
|
|
24
|
+
`settings.projectId`; it is not an OAuth registration or secret.
|
|
25
|
+
5. Connect the Google account that should read project metadata. Project IAM
|
|
26
|
+
access is separate from OAuth consent; the list filters to enabled projects
|
|
27
|
+
on which the caller has an applicable project-level role.
|
|
28
|
+
[Project-list requirements](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/projects/list).
|
|
29
|
+
|
|
30
|
+
## Runtime and AI composition
|
|
31
|
+
|
|
32
|
+
`projects.list` calls `GET https://bigquery.googleapis.com/bigquery/v2/projects`.
|
|
33
|
+
It accepts `maxResults` (1–50, default 50 in this fragment) and optional
|
|
34
|
+
`pageToken`. Keep following `nextPageToken` even if a page is shorter than
|
|
35
|
+
requested: server filtering can shorten pages. Results retain `totalItems`
|
|
36
|
+
and project references; `projects` may be absent when `totalItems` is zero.
|
|
37
|
+
[Endpoint](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/projects/list).
|
|
38
|
+
|
|
39
|
+
The form also exposes broader documented scopes, including Cloud read-only
|
|
40
|
+
and full access. Query operations require `bigquery` or `cloud-platform`. Compose
|
|
41
|
+
the [connection pattern](../patterns/api-key-connection/PATTERN.md) with the
|
|
42
|
+
file store and OAuth `beginAuthorization` / `completeAuthorization`; verification
|
|
43
|
+
input is empty. The application owns callback routing and access policy.
|
|
44
|
+
Service-account keys, ADC and workload identity are separate authentication
|
|
45
|
+
options, not implemented by this delegated-OAuth fragment.
|
|
46
|
+
[Authentication options](https://docs.cloud.google.com/bigquery/docs/authentication).
|
|
47
|
+
|
|
48
|
+
`jobs.query` submits GoogleSQL to `settings.projectId`. Input includes `query`,
|
|
49
|
+
optional `location`, `maximumBytesBilled` and `requestId`; `maxResults` defaults
|
|
50
|
+
to 100 (maximum 1000), and `timeoutMs` to 1000 (maximum 10000). This operation
|
|
51
|
+
can incur charges and perform writes permitted by IAM. The application's
|
|
52
|
+
backend must authorize its use and supply trusted SQL; never concatenate user
|
|
53
|
+
input into SQL. Pass up to 100 named scalar `parameters`, each with `name`,
|
|
54
|
+
`type` and a string `value`. For example, SQL `SELECT @name` takes
|
|
55
|
+
`parameters: [{ name: "name", type: "STRING", value: userInput }]`.
|
|
56
|
+
Values are sent separately from SQL using Google's named parameter format.
|
|
57
|
+
Array, struct and null parameters are not currently supported.
|
|
58
|
+
[Parameterized queries](https://docs.cloud.google.com/bigquery/docs/parameterized-queries).
|
|
59
|
+
|
|
60
|
+
If `jobComplete` is false, call `jobs.getQueryResults` with the returned `jobId`
|
|
61
|
+
and location. Pass `pageToken` for subsequent pages. Both operations use the
|
|
62
|
+
configured project; callers cannot override it. Results remain provider-shaped,
|
|
63
|
+
including warnings/errors for application handling. SQL and query-result page
|
|
64
|
+
tokens preserve whitespace. Rows retain native cells, nested records/repeated
|
|
65
|
+
values, nulls and decimal/large-integer strings; do not coerce them to JavaScript
|
|
66
|
+
numbers. Structural response validation rejects malformed row containers. There is no automatic retry
|
|
67
|
+
or polling, dry-run operation, or job ownership registry here.
|
|
68
|
+
The application must restrict job access within its own authorization policy.
|
|
69
|
+
[Query API](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query),
|
|
70
|
+
[result API](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/jobs/getQueryResults).
|
|
71
|
+
|
|
72
|
+
## Dataset, table and job operations
|
|
73
|
+
|
|
74
|
+
All operations below use `settings.projectId`; operation inputs cannot override
|
|
75
|
+
it. Use another project-owned configuration slot for metadata in another project.
|
|
76
|
+
Discovery requires the corresponding dataset/table IAM access independently of
|
|
77
|
+
OAuth consent. Neither discovery nor connection verification submits SQL.
|
|
78
|
+
|
|
79
|
+
| Operation | Inputs | Returned data |
|
|
80
|
+
|---|---|---|
|
|
81
|
+
| `datasets.list` | Optional `maxResults` 1–1000 (default 100), `pageToken` | Native dataset page |
|
|
82
|
+
| `tables.list` | `datasetId`; optional `maxResults`, `pageToken` | Native table page |
|
|
83
|
+
| `tables.get` | `datasetId`, `tableId` | Table metadata and schema, not table rows |
|
|
84
|
+
| `jobs.get` | `jobId`; optional `location` | Job status, statistics and error details |
|
|
85
|
+
| `jobs.cancel` | `jobId`; optional `location` | Cancellation response containing job metadata |
|
|
86
|
+
|
|
87
|
+
Metadata listing preserves references and opaque pagination tokens. Empty lists
|
|
88
|
+
can omit their collection field. Table schemas retain provider field types and
|
|
89
|
+
nested structure. Use query results to obtain data; fetching table metadata does
|
|
90
|
+
not fetch its rows. Native framework consumers can use the same
|
|
91
|
+
[datasets.list](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/datasets/list),
|
|
92
|
+
[tables.list](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tables/list)
|
|
93
|
+
and [tables.get](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/tables/get)
|
|
94
|
+
methods with their own Google libraries.
|
|
95
|
+
|
|
96
|
+
The application must authorize each job ID before inspection or cancellation.
|
|
97
|
+
Retain its region and use it when calling these operations. `jobs.get` preserves
|
|
98
|
+
PENDING, RUNNING and DONE, including `status.errorResult`: DONE does not mean
|
|
99
|
+
success. Job metadata can contain SQL and errors, so do not expose it to other
|
|
100
|
+
users. See [jobs.get](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get).
|
|
101
|
+
|
|
102
|
+
`jobs.cancel` requests cancellation; it does not prove the job stopped or that
|
|
103
|
+
charges were avoided. Inspect status afterwards. A local abort/disconnect does
|
|
104
|
+
not cancel the remote job. The cancellation request is not automatically retried.
|
|
105
|
+
See [jobs.cancel](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/jobs/cancel).
|
|
106
|
+
|
|
107
|
+
## Federation ownership — not implemented yet
|
|
108
|
+
|
|
109
|
+
Federation needs a trusted external identity as well as a workload identity
|
|
110
|
+
provider audience and a service-account email. Those two settings alone cannot
|
|
111
|
+
produce credentials. The generated application's deployment must supply the
|
|
112
|
+
identity; an editor login, project URL or OAuth client ID is not that identity.
|
|
113
|
+
|
|
114
|
+
For a deployment with an OIDC issuer, its operator configures a Google workload
|
|
115
|
+
identity pool/provider, grants that identity permission to impersonate the
|
|
116
|
+
service account, and generates Google's external-account credential
|
|
117
|
+
configuration. The Node application can use Google's authentication library;
|
|
118
|
+
other frameworks use their supported Google authentication tools. File-sourced
|
|
119
|
+
OIDC tokens require the deployment to update the token file before expiration.
|
|
120
|
+
[Google authentication library federation setup](https://github.com/googleapis/google-auth-library-nodejs#workload-identity-federation).
|
|
121
|
+
|
|
122
|
+
Vibe64 should edit the application configuration and Env references. It must
|
|
123
|
+
not issue deployment identities or require an exported application to call a
|
|
124
|
+
Vibe64 token gateway. The runtime implementation still needs to define and
|
|
125
|
+
validate its credential-source input and prove exchange and renewal before this
|
|
126
|
+
mode can be offered as working.
|
|
127
|
+
|
|
128
|
+
## Automation and application ownership
|
|
129
|
+
|
|
130
|
+
An authorized AI can prepare configuration, enable the API and assist with IAM
|
|
131
|
+
using existing Google administration APIs or gcloud. Web-client registration,
|
|
132
|
+
branding/review and account consent follow the common console guide.
|
|
133
|
+
The application owner supplies its OAuth registration and data-project access.
|
|
134
|
+
Separate OAuth projects do not automatically separate compute billing or resource
|
|
135
|
+
limits. Query submission uses the configured billing/execution project.
|
|
136
|
+
|
|
137
|
+
Automated tests cover consent, scopes, replay, private-file restart, isolation,
|
|
138
|
+
pagination, empty pages and provider failures without live queries or projects.
|
|
139
|
+
|
|
140
|
+
## Existing-scope closeout — 13 September 2026
|
|
141
|
+
|
|
142
|
+
Project-owned delegated Google OAuth supports project/dataset/table discovery, table metadata, named scalar GoogleSQL parameters, query submission, job status, result pagination and cancellation in the configured execution project.
|
|
143
|
+
|
|
144
|
+
No service-account key, ADC, workload-identity federation or deployment identity issuance. No array/struct/null parameters, automatic retry/polling/pagination, dry-run operation, load/extract/copy/streaming jobs, dedicated dataset/table administration or cross-project input override. SQL may incur charges and perform IAM-permitted writes; maximumBytesBilled is optional, not an account budget. Connection checking lists projects and does not prove query/dataset permissions. The app owns SQL authorization, per-user job ownership, result rendering, billing choices and cancellation follow-up; DONE or a cancellation acknowledgement does not prove successful execution/stopping. To extend: implement deployment-owned federation and its renewal/tests, then only required job/parameter modes. No Vibe64 identity gateway. No live provider account, provider registration, paid request or generated-application execution was tested. No new editor coding-assistant tool attachment is claimed. Other frameworks use the same project configuration and their own native tools; JSKIT is optional.
|
|
145
|
+
|
|
146
|
+
9 source and 9 installed-package BigQuery checks passed on September 13. September 12 compact/expanded configuration, IAM guidance and lifecycle browser evidence is retained; no new UI run.
|