@superglue/cli 1.1.21 → 1.1.23
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/README.md +1 -7
- package/dist/cli.js +307 -77
- package/dist/skills/superglue/SKILL.md +315 -631
- package/dist/skills/superglue/references/file-handling.md +180 -96
- package/dist/skills/superglue/references/graphql.md +131 -0
- package/dist/skills/superglue/references/http.md +251 -0
- package/dist/skills/superglue/references/integration.md +4 -2
- package/dist/skills/superglue/references/mssql.md +207 -0
- package/dist/skills/superglue/references/odbc.md +162 -0
- package/dist/skills/superglue/references/postgres.md +154 -0
- package/dist/skills/superglue/references/redis.md +68 -118
- package/dist/skills/superglue/references/sftp-smb.md +161 -0
- package/dist/skills/superglue/references/superglue-info.md +91 -0
- package/package.json +1 -1
- package/skills/superglue/SKILL.md +315 -631
- package/skills/superglue/references/file-handling.md +180 -96
- package/skills/superglue/references/graphql.md +131 -0
- package/skills/superglue/references/http.md +251 -0
- package/skills/superglue/references/integration.md +4 -2
- package/skills/superglue/references/mssql.md +207 -0
- package/skills/superglue/references/odbc.md +162 -0
- package/skills/superglue/references/postgres.md +154 -0
- package/skills/superglue/references/redis.md +68 -118
- package/skills/superglue/references/sftp-smb.md +161 -0
- package/skills/superglue/references/superglue-info.md +91 -0
- package/dist/skills/superglue/references/data-handling.md +0 -303
- package/dist/skills/superglue/references/databases.md +0 -253
- package/dist/skills/superglue/references/file-servers.md +0 -112
- package/dist/skills/superglue/references/http-apis.md +0 -248
- package/skills/superglue/references/data-handling.md +0 -303
- package/skills/superglue/references/databases.md +0 -253
- package/skills/superglue/references/file-servers.md +0 -112
- package/skills/superglue/references/http-apis.md +0 -248
|
@@ -0,0 +1,251 @@
|
|
|
1
|
+
# HTTP
|
|
2
|
+
|
|
3
|
+
## Step Configuration
|
|
4
|
+
|
|
5
|
+
```typescript
|
|
6
|
+
{
|
|
7
|
+
type: "request",
|
|
8
|
+
systemId: "my_api",
|
|
9
|
+
url: "<<my_api_url>>/v1/users",
|
|
10
|
+
method: "GET",
|
|
11
|
+
headers: { "Authorization": "Bearer <<my_api_access_token>>" },
|
|
12
|
+
queryParams: { "limit": "<<limit>>", "status": "active" },
|
|
13
|
+
body: '{"name": "<<(sourceData) => sourceData.currentItem.name>>"}',
|
|
14
|
+
pagination: { ... }
|
|
15
|
+
}
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
All fields support `<<variable>>` and `<<(sourceData) => ...>>` expressions.
|
|
19
|
+
|
|
20
|
+
### Fields
|
|
21
|
+
|
|
22
|
+
| Field | Required | Notes |
|
|
23
|
+
| ------------- | -------- | ------------------------------------------------------- |
|
|
24
|
+
| `url` | yes | Prefer `<<systemId_url>>/path` over hardcoded base URLs |
|
|
25
|
+
| `method` | yes | `GET`, `POST`, `PUT`, `DELETE`, `PATCH` |
|
|
26
|
+
| `headers` | no | Auth headers, content type, custom headers |
|
|
27
|
+
| `queryParams` | no | Key-value pairs appended to URL |
|
|
28
|
+
| `body` | no | Stripped for GET/HEAD/DELETE/OPTIONS |
|
|
29
|
+
| `pagination` | no | See Pagination section |
|
|
30
|
+
| `systemId` | no | Links system credentials and URL. Omit for public APIs |
|
|
31
|
+
|
|
32
|
+
## Authentication
|
|
33
|
+
|
|
34
|
+
Credentials are never injected automatically. You must place auth headers explicitly using `<<systemId_credentialKey>>` syntax.
|
|
35
|
+
|
|
36
|
+
```javascript
|
|
37
|
+
// Bearer token (including OAuth — token refresh is automatic, header is not)
|
|
38
|
+
{ "Authorization": "Bearer <<systemId_access_token>>" }
|
|
39
|
+
|
|
40
|
+
// API key
|
|
41
|
+
{ "X-API-Key": "<<systemId_api_key>>" }
|
|
42
|
+
|
|
43
|
+
// Basic Auth — auto-encoded to Base64, do NOT manually encode
|
|
44
|
+
{ "Authorization": "Basic <<systemId_username>>:<<systemId_password>>" }
|
|
45
|
+
|
|
46
|
+
// Runtime credentials from payload
|
|
47
|
+
{ "Authorization": "Bearer <<(sourceData) => sourceData.user_access_token>>" }
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
### SOAP Authentication
|
|
51
|
+
|
|
52
|
+
Some APIs (e.g. Salesforce) use SOAP XML login where credentials are embedded in the request body, not in headers. Multiple `<<>>` placeholders can appear adjacent — each is resolved independently and the results are concatenated.
|
|
53
|
+
|
|
54
|
+
```typescript
|
|
55
|
+
{
|
|
56
|
+
method: "POST",
|
|
57
|
+
url: "https://test.salesforce.com/services/Soap/u/59.0",
|
|
58
|
+
headers: {
|
|
59
|
+
"Content-Type": "text/xml; charset=UTF-8",
|
|
60
|
+
"SOAPAction": "login"
|
|
61
|
+
},
|
|
62
|
+
body: '<?xml version="1.0" encoding="utf-8"?><env:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"><env:Body><n1:login xmlns:n1="urn:partner.soap.sforce.com"><n1:username><<systemId_username>></n1:username><n1:password><<systemId_password>><<systemId_security_token>></n1:password></n1:login></env:Body></env:Envelope>'
|
|
63
|
+
}
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
`<<systemId_password>><<systemId_security_token>>` resolves to the password and token concatenated — no separator needed. This works in URL, headers, and body fields.
|
|
67
|
+
|
|
68
|
+
Headers starting with `x-` are treated as custom headers.
|
|
69
|
+
Modern APIs expect auth in headers, not query parameters, unless docs explicitly say otherwise.
|
|
70
|
+
|
|
71
|
+
## HTTP Runtime Details
|
|
72
|
+
|
|
73
|
+
### Request Behavior
|
|
74
|
+
|
|
75
|
+
- **GET, HEAD, DELETE, OPTIONS**: body is always stripped
|
|
76
|
+
- **POST, PUT, PATCH**: body with leading `{` is JSON-parsed; empty body becomes undefined
|
|
77
|
+
- Default headers: `Accept: */*`, Chrome-like `User-Agent`
|
|
78
|
+
- HTTPS: `rejectUnauthorized: false` (accepts self-signed certs)
|
|
79
|
+
|
|
80
|
+
### Body Resolution
|
|
81
|
+
|
|
82
|
+
The runtime processes the body in this order:
|
|
83
|
+
|
|
84
|
+
1. `<<>>` template variables are resolved — object/array returns are `JSON.stringify`'d before splicing
|
|
85
|
+
2. The resulting string is JSON-parsed when possible
|
|
86
|
+
3. `file::` references are resolved on the parsed object
|
|
87
|
+
4. Final send format is determined:
|
|
88
|
+
- raw bytes when the entire body is `file::<key>.raw`
|
|
89
|
+
- base64 string for `file::<key>.base64`
|
|
90
|
+
- `multipart/form-data` when the `Content-Type` header requests it and body resolves to an object
|
|
91
|
+
- `JSON.stringify(...)` for objects
|
|
92
|
+
- plain string for everything else
|
|
93
|
+
|
|
94
|
+
File references resolve only in the body, not in headers, query params, or URLs.
|
|
95
|
+
|
|
96
|
+
**Double-encoding risk:** the runtime stringifies at both step 1 (expression into template) and step 4 (object into fetch body). When the body contains nested JSON (e.g. LLM APIs), have `<<>>` expressions return a string via `JSON.stringify(...)`, or use a preceding transform step.
|
|
97
|
+
|
|
98
|
+
### File Uploads
|
|
99
|
+
|
|
100
|
+
**Raw body** — send a file as the entire HTTP body:
|
|
101
|
+
|
|
102
|
+
```typescript
|
|
103
|
+
{
|
|
104
|
+
method: "PUT",
|
|
105
|
+
headers: { "Content-Type": "application/pdf" },
|
|
106
|
+
body: "file::document.raw"
|
|
107
|
+
}
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
**Base64-in-JSON** — embed a base64-encoded file inside a structured payload:
|
|
111
|
+
|
|
112
|
+
```typescript
|
|
113
|
+
{
|
|
114
|
+
method: "POST",
|
|
115
|
+
headers: { "Content-Type": "application/json" },
|
|
116
|
+
body: '{"fileBase64":"file::document.base64"}'
|
|
117
|
+
}
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
**Multipart form** — set the `Content-Type` header and provide a JSON object of form fields:
|
|
121
|
+
|
|
122
|
+
```typescript
|
|
123
|
+
{
|
|
124
|
+
method: "POST",
|
|
125
|
+
headers: { "Content-Type": "multipart/form-data" },
|
|
126
|
+
body: '{"file":"file::report.raw","description":"monthly export"}'
|
|
127
|
+
}
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
The runtime builds a real `FormData` request: `file::...raw` values become file parts, scalars become text parts, non-file objects/arrays are JSON-stringified into text parts. The user-supplied `Content-Type` header is removed so `fetch` sets the boundary automatically.
|
|
131
|
+
|
|
132
|
+
Multipart restrictions:
|
|
133
|
+
|
|
134
|
+
- Body must resolve to a flat object of form fields
|
|
135
|
+
- Nested file refs inside objects are not supported
|
|
136
|
+
- Do not manually set a multipart boundary
|
|
137
|
+
- Use `.raw` for file parts, `.base64` for base64 text fields, `.extracted` for parsed content fields
|
|
138
|
+
|
|
139
|
+
### Response Handling
|
|
140
|
+
|
|
141
|
+
All HTTP responses are read as raw bytes, then classified by byte-level detection (see `file-handling.md` for the full detection logic).
|
|
142
|
+
|
|
143
|
+
- **Binary types** (PDF, Excel, DOCX, ZIP, GZIP): treated as files, parsed content also populates `data`. Later steps can reference via `file::<stepId>.raw/base64/extracted`.
|
|
144
|
+
- **`application/octet-stream`**: always treated as a file.
|
|
145
|
+
- **Responses over 25 MB**: treated as files as a safety fallback.
|
|
146
|
+
- **RAW + `Content-Disposition: attachment`**: treated as a file.
|
|
147
|
+
- **Structured/text responses** (JSON, XML, CSV, YAML, HTML, text under size limit): parsed inline to `data` only.
|
|
148
|
+
|
|
149
|
+
### Error Detection
|
|
150
|
+
|
|
151
|
+
Non-2xx responses throw with method, URL, response body preview (1000 chars), masked config, and retry count. Any response with status in the 2xx range is treated as success and returned to the step as-is — the runtime does not scan successful response bodies for error fields, so APIs that return HTTP 200 with `{ "error": "..." }` must be handled by your transform or by setting a `stopCondition`/assertion at the step level.
|
|
152
|
+
|
|
153
|
+
Bypass: set `failureBehavior: "continue"` to skip the non-2xx throw and return the response anyway.
|
|
154
|
+
|
|
155
|
+
### Retry Behavior
|
|
156
|
+
|
|
157
|
+
**Connection/server errors:**
|
|
158
|
+
|
|
159
|
+
- Default 5 retries, configurable via `retries` on the step
|
|
160
|
+
- Retries on fetch-level errors (network/DNS/abort except timeout) and on 5xx responses
|
|
161
|
+
- Fixed delay between retries (default 1000ms, configurable via `retryDelay`)
|
|
162
|
+
|
|
163
|
+
**Rate limiting (429):**
|
|
164
|
+
|
|
165
|
+
- Counts against the same retry budget as 5xx/network errors
|
|
166
|
+
- Respects `Retry-After` header (seconds or HTTP-date)
|
|
167
|
+
- Without header: exponential backoff `10^n * 1000ms + jitter` (capped at 1hr per wait)
|
|
168
|
+
- Total wait across all 429 retries capped at `MAX_RATE_LIMIT_WAIT_MS` (1hr)
|
|
169
|
+
|
|
170
|
+
**Timeouts:** Request timeout aborts the fetch and throws immediately — timeouts are not retried.
|
|
171
|
+
|
|
172
|
+
### Pagination
|
|
173
|
+
|
|
174
|
+
Makes multiple requests in a loop, merging results. Only configure after verifying the exact API pagination mechanism from docs.
|
|
175
|
+
|
|
176
|
+
**Configuration:**
|
|
177
|
+
|
|
178
|
+
```typescript
|
|
179
|
+
pagination: {
|
|
180
|
+
type: "offsetBased" | "pageBased" | "cursorBased",
|
|
181
|
+
pageSize: "50",
|
|
182
|
+
cursorPath: "meta.next_cursor", // cursorBased only — dot-path to cursor
|
|
183
|
+
stopCondition: "(response, pageInfo) => !response.data.meta.next_cursor"
|
|
184
|
+
}
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
**Variables** — the matching variable MUST appear in the request (URL, queryParams, headers, or body). Throws if missing.
|
|
188
|
+
|
|
189
|
+
| Type | Variable | Starts at | Increments by |
|
|
190
|
+
| ----------- | ---------------------------- | --------- | ------------------------ |
|
|
191
|
+
| pageBased | `<<page>>` | 1 | 1 |
|
|
192
|
+
| offsetBased | `<<offset>>` | 0 | pageSize |
|
|
193
|
+
| cursorBased | `<<cursor>>` | null | extracted via cursorPath |
|
|
194
|
+
| all | `<<limit>>` / `<<pageSize>>` | pageSize | — |
|
|
195
|
+
|
|
196
|
+
**Stop conditions** — JS expression evaluated in sandbox. Receives `(response, pageInfo)`:
|
|
197
|
+
|
|
198
|
+
- `response.data` = parsed API response body
|
|
199
|
+
- `response.headers` = response headers
|
|
200
|
+
- `pageInfo = { page, offset, cursor, totalFetched }`
|
|
201
|
+
- Return `true` to STOP
|
|
202
|
+
|
|
203
|
+
```javascript
|
|
204
|
+
"!response.data.meta.next_cursor";
|
|
205
|
+
"response.data.items.length === 0";
|
|
206
|
+
"response.data.hasMore === false";
|
|
207
|
+
"pageInfo.totalFetched >= 1000";
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
**Cursor extraction:** `cursorPath` is a simple dot-separated property path (e.g. `meta.next_cursor`, `data.0.id`) resolved against the parsed response body. It is not a JSONPath expression — wildcards, filters, and `$.` prefixes are not supported. For property names containing `@` (e.g. OData `@odata.nextLink`), use the name directly as the path. Null/undefined cursor stops pagination.
|
|
211
|
+
|
|
212
|
+
**Safety checks (when stopCondition is set):**
|
|
213
|
+
|
|
214
|
+
1. Identical pages 1 and 2 (both with data): throws — pagination params are not being applied
|
|
215
|
+
2. Both pages empty + stop did not trigger: throws — broken stop condition
|
|
216
|
+
3. Duplicate consecutive response after page 2: auto-stops
|
|
217
|
+
|
|
218
|
+
**Without stopCondition (fallback):**
|
|
219
|
+
|
|
220
|
+
- Array shorter than pageSize: stop
|
|
221
|
+
- Duplicate response hash against any previously seen page: stop
|
|
222
|
+
- Non-array response: stop after first request
|
|
223
|
+
- Max 500 requests (vs 1000 max with a stopCondition)
|
|
224
|
+
|
|
225
|
+
**Paginated step results:** all pages merge into a single result. Access via `sourceData.stepId.data` — it is NOT an array of per-page results. Do not call `.map()` on a paginated step result. Returns single object if one result, array if multiple (unwrapped from single-element array).
|
|
226
|
+
|
|
227
|
+
### File Output
|
|
228
|
+
|
|
229
|
+
When an HTTP step downloads a file (PDF, CSV, binary response), set `outputFile: true` on the step to make it downloadable by the user. Without the flag, downloaded files stay internal (only accessible to subsequent steps via `file::` references).
|
|
230
|
+
|
|
231
|
+
## Common Pitfalls
|
|
232
|
+
|
|
233
|
+
- Hardcoding base URLs that match the system's base URL instead of using `<<systemId_url>>`
|
|
234
|
+
- Forgetting auth headers entirely — nothing is injected automatically, including for OAuth systems (only token refresh is automatic)
|
|
235
|
+
- Forgetting `outputFile: true` on steps that download files the user wants
|
|
236
|
+
- Putting pagination variables in the config but not setting a `pagination` block, or vice versa
|
|
237
|
+
- Using `<<currentItem.id>>` — must use arrow function syntax: `<<(sourceData) => sourceData.currentItem.id>>`
|
|
238
|
+
- Mixing multiple `<<>>` expressions in one body string when the API expects nested JSON — use a transform step instead
|
|
239
|
+
- Setting `method: "POST"` for GraphQL queries without `modify: false` — POST that only reads should not be marked as modifying
|
|
240
|
+
|
|
241
|
+
## Error Recovery
|
|
242
|
+
|
|
243
|
+
When an HTTP step fails and the cause is not immediately clear:
|
|
244
|
+
|
|
245
|
+
1. **Isolate endpoint vs system-level issues** — use `sg system call` to test a different endpoint on the same system. If multiple endpoints fail, the issue is likely global authentication or system configuration. If only one fails, the issue is endpoint-specific (wrong URL, missing params, incorrect method or authentication scopes).
|
|
246
|
+
|
|
247
|
+
2. **Check credential scopes and system-side configuration** — a 403 or empty response often means the API key or OAuth token lacks required scopes, or the resource is restricted server-side. Use `sg system search-docs` or a web search to look up required scopes/permissions for the endpoint.
|
|
248
|
+
|
|
249
|
+
3. **Verify request shape against API docs** — use `sg system search-docs` and web search to confirm the endpoint URL, required headers, query parameters, body format, and expected method. APIs change across versions. Search for the error message and the endpoint in docs and online.
|
|
250
|
+
|
|
251
|
+
4. **Check for rate limiting** — 429 errors are retried automatically with backoff, but persistent 429s may mean the API key has hit a quota. Check API dashboard or docs for rate limit details.
|
|
@@ -73,12 +73,14 @@ curl -X POST "$SUPERGLUE_API_ENDPOINT/v1/runs/{runId}/cancel" \
|
|
|
73
73
|
|
|
74
74
|
## SDK Installation
|
|
75
75
|
|
|
76
|
+
**Always install the latest version** — older SDK versions may have known bugs (e.g. case-sensitivity issues on Linux in versions before 3.3.0).
|
|
77
|
+
|
|
76
78
|
```bash
|
|
77
79
|
# TypeScript/JavaScript
|
|
78
|
-
npm install @superglue/client
|
|
80
|
+
npm install @superglue/client@latest
|
|
79
81
|
|
|
80
82
|
# Python
|
|
81
|
-
pip install superglue-client
|
|
83
|
+
pip install --upgrade superglue-client
|
|
82
84
|
```
|
|
83
85
|
|
|
84
86
|
Both SDKs are auto-generated from the OpenAPI spec (JS via orval, Python via openapi-python-client).
|
|
@@ -0,0 +1,207 @@
|
|
|
1
|
+
# MSSQL
|
|
2
|
+
|
|
3
|
+
## Step Configuration
|
|
4
|
+
|
|
5
|
+
```typescript
|
|
6
|
+
{
|
|
7
|
+
type: "request",
|
|
8
|
+
systemId: "my_azure_sql",
|
|
9
|
+
url: "mssql://<<my_azure_sql_user>>:<<my_azure_sql_password>>@<<my_azure_sql_host>>:<<my_azure_sql_port>>/<<my_azure_sql_database>>",
|
|
10
|
+
body: '{"query": "SELECT * FROM users WHERE id = @param1", "params": [<<userId>>]}'
|
|
11
|
+
}
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Configure the connection URL based on which credentials are stored in the system — use `sg system find` to check `storedCredentials` and reference them accordingly.
|
|
15
|
+
|
|
16
|
+
`method`, `headers`, `queryParams`, and `pagination` are HTTP-only fields — omit them for database steps.
|
|
17
|
+
|
|
18
|
+
### Fields
|
|
19
|
+
|
|
20
|
+
| Field | Required | Notes |
|
|
21
|
+
| ---------- | -------- | -------------------------------------------------------------- |
|
|
22
|
+
| `url` | yes | Connection string with `<<credential>>` placeholders |
|
|
23
|
+
| `body` | yes | JSON string with `query` and optional `params` |
|
|
24
|
+
| `systemId` | no | Links system credentials for `<<systemId_credKey>>` resolution |
|
|
25
|
+
|
|
26
|
+
### URL Format
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
mssql://user:password@host:port/database
|
|
30
|
+
sqlserver://user:password@host:port/database
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Both `mssql://` and `sqlserver://` are supported.
|
|
34
|
+
|
|
35
|
+
**Azure SQL Database** — use the fully qualified server name:
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
mssql://myuser:mypassword@myserver.database.windows.net:1433/mydatabase
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Azure SQL usernames often include `@servername` suffix (e.g., `myuser@myserver`). Encode the `@` as `%40` in the URL:
|
|
42
|
+
|
|
43
|
+
```
|
|
44
|
+
mssql://myuser%40myserver:mypassword@myserver.database.windows.net:1433/mydatabase
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Connection parameters** can be appended as query strings:
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
mssql://user:password@host:port/database?encrypt=true&trustServerCertificate=false
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
| Parameter | Default | Notes |
|
|
54
|
+
| ------------------------ | ------- | --------------------------------------- |
|
|
55
|
+
| `encrypt` | `true` | Required for Azure SQL |
|
|
56
|
+
| `trustServerCertificate` | `false` | Whether to trust the server certificate |
|
|
57
|
+
| `database` | — | Can be in URL path or query parameter |
|
|
58
|
+
|
|
59
|
+
### Body Format
|
|
60
|
+
|
|
61
|
+
JSON string with `query` and optional `params` (or `values`):
|
|
62
|
+
|
|
63
|
+
```json
|
|
64
|
+
{
|
|
65
|
+
"query": "SELECT * FROM users WHERE age > @param1 AND status = @param2",
|
|
66
|
+
"params": [25, "active"]
|
|
67
|
+
}
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
## Authentication
|
|
71
|
+
|
|
72
|
+
### SQL Server Authentication (default)
|
|
73
|
+
|
|
74
|
+
Credentials are embedded directly in the connection URL using `<<systemId_credentialKey>>` placeholders. Nothing is injected automatically.
|
|
75
|
+
|
|
76
|
+
```
|
|
77
|
+
mssql://<<my_db_user>>:<<my_db_password>>@<<my_db_host>>:<<my_db_port>>/<<my_db_database>>
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### Azure Active Directory Authentication
|
|
81
|
+
|
|
82
|
+
For Azure AD auth, store the following credential keys on the system alongside the standard connection fields:
|
|
83
|
+
|
|
84
|
+
**Entra ID User + Password** — authenticates as an Azure AD user. Requires an Azure app registration.
|
|
85
|
+
|
|
86
|
+
| Credential Key | Required | Notes |
|
|
87
|
+
| -------------- | -------- | ------------------------------------------------------ |
|
|
88
|
+
| `authType` | yes | `azure-active-directory-password` |
|
|
89
|
+
| `clientId` | yes | Application (client) ID from Azure app registration |
|
|
90
|
+
| `tenantId` | no | Directory (tenant) ID |
|
|
91
|
+
| `user` | yes | Azure AD username (also accepts `username`, or in URL) |
|
|
92
|
+
| `password` | yes | Azure AD password (or in URL) |
|
|
93
|
+
|
|
94
|
+
`user` and `password` can be stored as credentials or embedded in the URL — credentials take precedence.
|
|
95
|
+
|
|
96
|
+
```
|
|
97
|
+
URL: mssql://<<sys_user>>:<<sys_password>>@myserver.database.windows.net:1433/mydb
|
|
98
|
+
Credentials: { authType, clientId, tenantId, user, password, host, port, database }
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
**Service Principal** — authenticates as an Azure AD application. No user/password needed in the URL.
|
|
102
|
+
|
|
103
|
+
| Credential Key | Required | Notes |
|
|
104
|
+
| -------------- | -------- | ------------------------------------------------- |
|
|
105
|
+
| `authType` | yes | `azure-active-directory-service-principal-secret` |
|
|
106
|
+
| `clientId` | yes | Application (client) ID |
|
|
107
|
+
| `clientSecret` | yes | Client secret |
|
|
108
|
+
| `tenantId` | yes | Directory (tenant) ID |
|
|
109
|
+
|
|
110
|
+
```
|
|
111
|
+
URL: mssql://myserver.database.windows.net:1433/mydb
|
|
112
|
+
Credentials: { authType, clientId, clientSecret, tenantId, host, port, database }
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The runtime authenticates entirely via the service principal credentials — no user or password is needed in the URL.
|
|
116
|
+
|
|
117
|
+
## MSSQL Runtime Details
|
|
118
|
+
|
|
119
|
+
### Schema Discovery
|
|
120
|
+
|
|
121
|
+
superglue does not auto-introspect MSSQL schemas. Before building tools, always query the schema manually with `sg system call` to discover tables and columns:
|
|
122
|
+
|
|
123
|
+
```json
|
|
124
|
+
{
|
|
125
|
+
"query": "SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE, IS_NULLABLE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = 'dbo' ORDER BY TABLE_NAME, ORDINAL_POSITION"
|
|
126
|
+
}
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
For Azure SQL or databases with non-default schemas, adjust the `TABLE_SCHEMA` filter or remove it to see all schemas.
|
|
130
|
+
Always verify table and column names against the actual schema before building tools.
|
|
131
|
+
|
|
132
|
+
### Parameterized Queries
|
|
133
|
+
|
|
134
|
+
Always use parameterized queries with `@param1`, `@param2`, etc. This prevents SQL injection. The params array maps positionally: `params[0]` maps to `@param1`, `params[1]` maps to `@param2`, etc.
|
|
135
|
+
|
|
136
|
+
```json
|
|
137
|
+
{ "query": "SELECT * FROM orders WHERE customer_id = @param1", "params": ["<<customerId>>"] }
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
With expressions:
|
|
141
|
+
|
|
142
|
+
```json
|
|
143
|
+
{
|
|
144
|
+
"query": "INSERT INTO logs (message, level) VALUES (@param1, @param2)",
|
|
145
|
+
"params": ["<<(sourceData) => sourceData.message>>", "error"]
|
|
146
|
+
}
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
With OUTPUT (MSSQL equivalent of RETURNING):
|
|
150
|
+
|
|
151
|
+
```json
|
|
152
|
+
{
|
|
153
|
+
"query": "INSERT INTO users (name, email) OUTPUT INSERTED.* VALUES (@param1, @param2)",
|
|
154
|
+
"params": [
|
|
155
|
+
"<<(sourceData) => sourceData.currentItem.name>>",
|
|
156
|
+
"<<(sourceData) => sourceData.currentItem.email>>"
|
|
157
|
+
]
|
|
158
|
+
}
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
### Return Value
|
|
162
|
+
|
|
163
|
+
Returns the recordset — array of row objects with column names as keys.
|
|
164
|
+
|
|
165
|
+
- `SELECT`: array of matching rows
|
|
166
|
+
- `INSERT/UPDATE/DELETE` with `OUTPUT`: array of returned rows
|
|
167
|
+
- Mutations without `OUTPUT`: empty array
|
|
168
|
+
|
|
169
|
+
### Batch Operations
|
|
170
|
+
|
|
171
|
+
Use a data selector returning an array to execute a query per item:
|
|
172
|
+
|
|
173
|
+
```javascript
|
|
174
|
+
// dataSelector: (sourceData) => sourceData.newUsers
|
|
175
|
+
// body: {"query": "INSERT INTO users (name) VALUES (@param1)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Connection Pooling
|
|
179
|
+
|
|
180
|
+
- Pools cached by connection string, max 10 clients per pool
|
|
181
|
+
- 30s connection timeout, configurable request timeout
|
|
182
|
+
- Idle pools cleaned up automatically
|
|
183
|
+
- SSL/TLS encryption enabled by default for Azure SQL
|
|
184
|
+
|
|
185
|
+
### Retry Behavior
|
|
186
|
+
|
|
187
|
+
Default retries from server config. On final failure, error includes query text and params.
|
|
188
|
+
|
|
189
|
+
## Common Pitfalls
|
|
190
|
+
|
|
191
|
+
- Using `$1`, `$2` (PostgreSQL syntax) instead of `@param1`, `@param2` — MSSQL uses named `@param` parameters
|
|
192
|
+
- Forgetting `OUTPUT INSERTED.*` on INSERT/UPDATE/DELETE when you need the result — without it the step returns an empty array
|
|
193
|
+
- Accessing step result directly instead of via `.data` — database results follow the same envelope as all steps: `sourceData.stepId.data`
|
|
194
|
+
- Not URL-encoding `@` in Azure SQL usernames — `myuser@myserver` must be `myuser%40myserver` in the connection URL
|
|
195
|
+
|
|
196
|
+
## Error Recovery
|
|
197
|
+
|
|
198
|
+
When an MSSQL step fails and the cause is not obvious:
|
|
199
|
+
|
|
200
|
+
1. **Read the error message** — superglue includes the query text and params in the error. Check for syntax errors, missing columns, or type mismatches.
|
|
201
|
+
2. **Verify connection credentials** — use `sg system find` to confirm `storedCredentials` keys match the placeholders in your URL. Test connectivity with a simple `SELECT 1` via `sg system call`.
|
|
202
|
+
3. **Check database-side access** — connection refused or permission denied errors typically mean the database user lacks access. Verify with the user that the credentials have the required grants.
|
|
203
|
+
4. **Check Azure-specific issues** — if connecting to Azure SQL:
|
|
204
|
+
- Confirm the client IP is allowed in the Azure SQL firewall rules
|
|
205
|
+
- Verify the username includes the `@servername` suffix if required, properly encoded as `%40`
|
|
206
|
+
- Confirm `encrypt=true` is set (required for Azure SQL)
|
|
207
|
+
5. **Verify parameter types** — MSSQL infers types from the values passed. Passing a string where a number is expected may cause implicit conversion errors. Ensure `params` values match expected column types.
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
# ODBC
|
|
2
|
+
|
|
3
|
+
## Step Configuration
|
|
4
|
+
|
|
5
|
+
```typescript
|
|
6
|
+
{
|
|
7
|
+
type: "request",
|
|
8
|
+
systemId: "my_sybase_db",
|
|
9
|
+
url: "odbc://<<my_sybase_db_username>>:<<my_sybase_db_password>>@<<my_sybase_db_host>>:<<my_sybase_db_port>>/<<my_sybase_db_database>>?DRIVER=<<my_sybase_db_driver>>",
|
|
10
|
+
body: '{"query": "SELECT * FROM users WHERE id = ?", "params": [<<userId>>]}'
|
|
11
|
+
}
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Configure the connection URL based on which credentials are stored in the system — use `sg system find` to check `storedCredentials` and reference them accordingly.
|
|
15
|
+
|
|
16
|
+
`method`, `headers`, `queryParams`, and `pagination` are HTTP-only fields — omit them for database steps.
|
|
17
|
+
|
|
18
|
+
### Fields
|
|
19
|
+
|
|
20
|
+
| Field | Required | Notes |
|
|
21
|
+
| ---------- | -------- | -------------------------------------------------------------- |
|
|
22
|
+
| `url` | yes | Connection string with `<<credential>>` placeholders |
|
|
23
|
+
| `body` | yes | JSON string with `query` and optional `params` |
|
|
24
|
+
| `systemId` | no | Links system credentials for `<<systemId_credKey>>` resolution |
|
|
25
|
+
|
|
26
|
+
### URL Format
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
odbc://user:password@host:port/database?DRIVER=DriverName
|
|
30
|
+
odbc://user:password@host:port/database?DRIVER=FreeTDS&TDS_Version=5.0
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
The path contains the **database name**. The ODBC driver is specified via the `DRIVER` query parameter (defaults to `FreeTDS` if omitted). This format is consistent with other superglue database strategies (postgres, mssql) where the path is always the database.
|
|
34
|
+
|
|
35
|
+
**Supported drivers:**
|
|
36
|
+
|
|
37
|
+
| Driver | Database | Default Port |
|
|
38
|
+
| --------- | ---------------------------- | ------------ |
|
|
39
|
+
| `FreeTDS` | SAP ASE (Sybase), SQL Server | 5000 |
|
|
40
|
+
|
|
41
|
+
Additional query parameters (beyond `DRIVER`) are passed through to the ODBC connection string as key=value pairs.
|
|
42
|
+
|
|
43
|
+
### Body Format
|
|
44
|
+
|
|
45
|
+
JSON string with `query` and optional `params` (or `values`):
|
|
46
|
+
|
|
47
|
+
```json
|
|
48
|
+
{
|
|
49
|
+
"query": "SELECT * FROM users WHERE age > ? AND status = ?",
|
|
50
|
+
"params": [25, "active"]
|
|
51
|
+
}
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## Authentication
|
|
55
|
+
|
|
56
|
+
Credentials are embedded directly in the connection URL using `<<systemId_credentialKey>>` placeholders. Nothing is injected automatically.
|
|
57
|
+
|
|
58
|
+
```
|
|
59
|
+
odbc://<<my_db_username>>:<<my_db_password>>@<<my_db_host>>:<<my_db_port>>/<<my_db_database>>?DRIVER=<<my_db_driver>>
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Credential variables are resolved at runtime before connecting.
|
|
63
|
+
|
|
64
|
+
## ODBC Runtime Details
|
|
65
|
+
|
|
66
|
+
### Schema Discovery
|
|
67
|
+
|
|
68
|
+
superglue does not auto-introspect ODBC database schemas. Before building tools, always query the schema manually with `sg system call` to discover tables and columns.
|
|
69
|
+
|
|
70
|
+
**SAP ASE (Sybase):**
|
|
71
|
+
|
|
72
|
+
```json
|
|
73
|
+
{
|
|
74
|
+
"query": "SELECT o.name AS table_name, c.name AS column_name, t.name AS data_type, CASE WHEN c.status & 8 = 8 THEN 'YES' ELSE 'NO' END AS is_nullable FROM sysobjects o JOIN syscolumns c ON o.id = c.id JOIN systypes t ON c.usertype = t.usertype WHERE o.type = 'U' ORDER BY o.name, c.colid"
|
|
75
|
+
}
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Always verify table and column names against the actual schema before building tools.
|
|
79
|
+
|
|
80
|
+
### Parameterized Queries
|
|
81
|
+
|
|
82
|
+
Always use parameterized queries with positional `?` placeholders — prevents SQL injection. Parameters are bound positionally from the `params` array.
|
|
83
|
+
|
|
84
|
+
For numeric or boolean parameters in query params arrays, use unquoted placeholders: `[<<numericParam>>]` not `["<<numericParam>>"]`. The quotes force string resolution.
|
|
85
|
+
|
|
86
|
+
```json
|
|
87
|
+
{ "query": "SELECT * FROM orders WHERE name = ? AND customer_id = ?", "params": ["<<customerName>>", <<customerId>>] }
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
With expressions:
|
|
91
|
+
|
|
92
|
+
```json
|
|
93
|
+
{
|
|
94
|
+
"query": "INSERT INTO logs (message, level) VALUES (?, ?)",
|
|
95
|
+
"params": ["<<(sourceData) => sourceData.message>>", "error"]
|
|
96
|
+
}
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### Return Value
|
|
100
|
+
|
|
101
|
+
Returns an array of row objects with column names as keys.
|
|
102
|
+
|
|
103
|
+
- `SELECT`: array of matching rows
|
|
104
|
+
- `INSERT/UPDATE/DELETE`: empty array (SAP ASE does not support `RETURNING` — query the data separately if needed)
|
|
105
|
+
|
|
106
|
+
### Batch Operations
|
|
107
|
+
|
|
108
|
+
Use a data selector returning an array to execute a query per item:
|
|
109
|
+
|
|
110
|
+
```javascript
|
|
111
|
+
// dataSelector: (sourceData) => sourceData.newUsers
|
|
112
|
+
// body: {"query": "INSERT INTO users (name) VALUES (?)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Retry Behavior
|
|
116
|
+
|
|
117
|
+
Default retries from server config. On final failure, error includes a truncated query preview.
|
|
118
|
+
|
|
119
|
+
## SAP ASE (Sybase) Specifics
|
|
120
|
+
|
|
121
|
+
### Differences from SQL Server
|
|
122
|
+
|
|
123
|
+
SAP ASE uses Transact-SQL like SQL Server, but with key differences:
|
|
124
|
+
|
|
125
|
+
- **No `OUTPUT` clause** — use `SELECT @@identity` after INSERT to get the last inserted identity value
|
|
126
|
+
- **No `TOP` with `ORDER BY` guarantee** — use `SET ROWCOUNT n` before the query instead
|
|
127
|
+
- **`CONVERT` syntax differs** — `CONVERT(target_type, expression)` not `CONVERT(expression, style)`
|
|
128
|
+
- **String concatenation** — use `+` operator, not `CONCAT()` (unless ASE 16.0+)
|
|
129
|
+
- **Date functions** — use `GETDATE()` not `CURRENT_TIMESTAMP`, `DATEADD`/`DATEDIFF` syntax matches SQL Server
|
|
130
|
+
|
|
131
|
+
### TDS Protocol Version
|
|
132
|
+
|
|
133
|
+
FreeTDS defaults to TDS 5.0 for Sybase connections. If you encounter protocol errors, you can specify the version via query parameter:
|
|
134
|
+
|
|
135
|
+
```
|
|
136
|
+
odbc://user:pass@host:5000/mydb?DRIVER=FreeTDS&TDS_Version=5.0
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
| TDS Version | Use Case |
|
|
140
|
+
| ----------- | -------------------------- |
|
|
141
|
+
| `5.0` | SAP ASE / Sybase (default) |
|
|
142
|
+
| `7.0` | SQL Server 7.0+ |
|
|
143
|
+
| `7.4` | SQL Server 2012+ |
|
|
144
|
+
|
|
145
|
+
## Common Pitfalls
|
|
146
|
+
|
|
147
|
+
- Using `@param1`, `@param2` (MSSQL syntax) or `$1`, `$2` (PostgreSQL syntax) instead of positional `?` placeholders — ODBC uses `?` for all parameter binding
|
|
148
|
+
- Forgetting that SAP ASE has no `OUTPUT`/`RETURNING` clause — mutations return empty arrays; query the data separately
|
|
149
|
+
- Using `CONCAT()` on older ASE versions — use `+` for string concatenation instead
|
|
150
|
+
- Driver name mismatch — the `DRIVER` query parameter must exactly match the section name in `odbcinst.ini` (case-sensitive). Use `DRIVER=FreeTDS`, not `DRIVER=freetds` or `DRIVER=FREETDS`
|
|
151
|
+
- Accessing step result directly instead of via `.data` — database results follow the same envelope as all steps: `sourceData.stepId.data`
|
|
152
|
+
|
|
153
|
+
## Error Recovery
|
|
154
|
+
|
|
155
|
+
When an ODBC step fails and the cause is not obvious:
|
|
156
|
+
|
|
157
|
+
1. **Read the error message** — superglue includes the a truncated query preview in the error. Check for syntax errors, missing columns, or type mismatches.
|
|
158
|
+
2. **Verify connection credentials** — use `sg system find` to confirm `storedCredentials` keys match the placeholders in your URL. Test connectivity with a simple `SELECT 1` via `sg system call`.
|
|
159
|
+
3. **Check database-side access** — connection refused or permission denied errors typically mean the database user lacks access. Verify with the user that the credentials have the required grants.
|
|
160
|
+
4. **Check driver availability** — if the error mentions "driver not found", the `DRIVER` query parameter does not match any registered driver in `/etc/odbcinst.ini`. Verify the driver name is correct and the driver is installed in the Docker image.
|
|
161
|
+
5. **Test the query directly** — use `sg system call` with a simplified version of the query to isolate whether the issue is the query itself, the parameters, or the connection.
|
|
162
|
+
6. **Check TDS version** — protocol errors may indicate a TDS version mismatch. Try adding `?TDS_Version=5.0` to the query parameters for SAP ASE connections.
|