@superglue/cli 1.1.20 → 1.1.22

package/dist/skills/superglue/references/databases.md

@@ -1,253 +0,0 @@
-# Database Steps
-
-Superglue supports both PostgreSQL and Microsoft SQL Server (including Azure SQL).
-
----
-
-## PostgreSQL
-
-### Step Configuration
-
-```typescript
-{
-  type: "request",
-  systemId: "my_postgres_db",
-  url: "postgres://<<my_postgres_db_user>>:<<my_postgres_db_password>>@<<my_postgres_db_host>>:<<my_postgres_db_port>>/<<my_postgres_db_database>>",
-  body: '{"query": "SELECT * FROM users WHERE id = $1", "params": [<<userId>>]}'
-}
-```
-
-Configure the connection url depending on which credentials are saved in the stored db system - reference them accordingly.
-
-## URL Format
-
-```
-postgres://user:password@host:port/database
-postgresql://user:password@host:port/database
-```
-
-All credential variables are resolved before connecting. Trailing slashes and extra slashes before query strings are cleaned.
-
-## Body Format
-
-JSON string with `query` and optional `params` (or `values`):
-
-```json
-{ "query": "SELECT * FROM users WHERE age > $1 AND status = $2", "params": [25, "active"] }
-```
-
-### Parameterized Queries
-
-**Always use parameterized queries** with `$1`, `$2`, etc. — prevents SQL injection.
-
-```json
-{ "query": "SELECT * FROM orders WHERE customer_id = $1", "params": ["<<customerId>>"] }
-```
-
-With expressions:
-
-```json
-{
-  "query": "INSERT INTO logs (message, level) VALUES ($1, $2)",
-  "params": ["<<(sourceData) => sourceData.message>>", "error"]
-}
-```
-
-With RETURNING:
-
-```json
-{
-  "query": "INSERT INTO users (name, email) VALUES ($1, $2) RETURNING *",
-  "params": [
-    "<<(sourceData) => sourceData.currentItem.name>>",
-    "<<(sourceData) => sourceData.currentItem.email>>"
-  ]
-}
-```
-
-## Connection Pooling
-
-- Pools cached by connection string, max 10 clients, 5s connection timeout
-- Idle pools cleaned up automatically
-- SSL auto-configured: enabled (with `rejectUnauthorized: false`) unless connecting to localhost without explicit sslmode
-
-## Retry Logic
-
-Default retries from server config. On final failure, error includes query text and params.
-
-## Return Value
-
-Returns `result.rows` — array of row objects with column names as keys.
-
-- `SELECT`: array of matching rows
-- `INSERT/UPDATE/DELETE` with `RETURNING`: array of returned rows
-- Mutations without `RETURNING`: empty array
-
-### Batch Operations via Loop
-
-Use a data selector returning an array to execute a query per item:
-
-```javascript
-// dataSelector: (sourceData) => sourceData.newUsers
-// body: {"query": "INSERT INTO users (name) VALUES ($1)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
-```
-
----
-
-## Microsoft SQL Server / Azure SQL
-
-### Step Configuration
-
-```typescript
-{
-  type: "request",
-  systemId: "my_azure_sql",
-  url: "mssql://<<my_azure_sql_user>>:<<my_azure_sql_password>>@<<my_azure_sql_host>>:<<my_azure_sql_port>>/<<my_azure_sql_database>>",
-  body: '{"query": "SELECT * FROM users WHERE id = @param1", "params": [<<userId>>]}'
-}
-```
-
-Configure the connection url depending on which credentials are saved in the stored db system - reference them accordingly.
-
-### URL Format
-
-```
-mssql://user:password@host:port/database
-sqlserver://user:password@host:port/database
-```
-
-Both `mssql://` and `sqlserver://` protocols are supported. All credential variables are resolved before connecting.
-
-#### Azure SQL Database Connection
-
-For Azure SQL Database, use the fully qualified server name:
-
-```
-mssql://myuser:mypassword@myserver.database.windows.net:1433/mydatabase
-```
-
-Note: Azure SQL usernames often include `@servername` suffix (e.g., `myuser@myserver`). When using in URLs, encode the @ as `%40`:
-
-```
-mssql://myuser%40myserver:mypassword@myserver.database.windows.net:1433/mydatabase
-```
-
-You can include connection parameters as query strings:
-
-```
-mssql://user:password@host:port/database?encrypt=true&trustServerCertificate=false
-```
-
-**Connection Parameters:**
-
-- `encrypt` (default: `true`) - Enable/disable encryption (required for Azure SQL)
-- `trustServerCertificate` (default: `false`) - Whether to trust the server certificate
-- `database` - Database name (can be in URL path or query parameter)
-
-### Body Format
-
-JSON string with `query` and optional `params` (or `values`):
-
-```json
-{
-  "query": "SELECT * FROM users WHERE age > @param1 AND status = @param2",
-  "params": [25, "active"]
-}
-```
-
-### Parameterized Queries
-
-**Always use parameterized queries** with `@param1`, `@param2`, etc. — prevents SQL injection.
-
-```json
-{ "query": "SELECT * FROM orders WHERE customer_id = @param1", "params": ["<<customerId>>"] }
-```
-
-With expressions:
-
-```json
-{
-  "query": "INSERT INTO logs (message, level) VALUES (@param1, @param2)",
-  "params": ["<<(sourceData) => sourceData.message>>", "error"]
-}
-```
-
-With OUTPUT (MSSQL equivalent of RETURNING):
-
-```json
-{
-  "query": "INSERT INTO users (name, email) OUTPUT INSERTED.* VALUES (@param1, @param2)",
-  "params": [
-    "<<(sourceData) => sourceData.currentItem.name>>",
-    "<<(sourceData) => sourceData.currentItem.email>>"
-  ]
-}
-```
-
-**Parameter Naming**: MSSQL uses named parameters with `@` prefix. The params array maps positionally: `params[0]` → `@param1`, `params[1]` → `@param2`, etc.
-
-### Connection Pooling
-
-- Pools cached by connection string, max 10 clients per pool
-- 5s connection timeout, configurable request timeout
-- Idle pools cleaned up automatically
-- SSL/TLS encryption enabled by default for Azure SQL
-
-### Retry Logic
-
-Default retries from server config. On final failure, error includes query text and params.
-
-### Return Value
-
-Returns the recordset — array of row objects with column names as keys.
-
-- `SELECT`: array of matching rows
-- `INSERT/UPDATE/DELETE` with `OUTPUT`: array of returned rows
-- Mutations without `OUTPUT`: empty array
-
-### Batch Operations via Loop
-
-Use a data selector returning an array to execute a query per item:
-
-```javascript
-// dataSelector: (sourceData) => sourceData.newUsers
-// body: {"query": "INSERT INTO users (name) VALUES (@param1)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
-```
-
-### Common Patterns
-
-**Simple SELECT:**
-
-```json
-{
-  "query": "SELECT id, name, email FROM users WHERE status = @param1",
-  "params": ["active"]
-}
-```
-
-**INSERT with OUTPUT:**
-
-```json
-{
-  "query": "INSERT INTO users (name, email, created_at) OUTPUT INSERTED.id, INSERTED.created_at VALUES (@param1, @param2, GETUTCDATE())",
-  "params": ["John Doe", "john@example.com"]
-}
-```
-
-**UPDATE with OUTPUT:**
-
-```json
-{
-  "query": "UPDATE users SET last_login = GETUTCDATE() OUTPUT INSERTED.id, INSERTED.last_login WHERE id = @param1",
-  "params": [123]
-}
-```
-
-**Stored Procedure Call:**
-
-```json
-{
-  "query": "EXEC dbo.GetUserById @param1",
-  "params": [123]
-}
-```

package/dist/skills/superglue/references/file-servers.md

@@ -1,112 +0,0 @@
-# File Server Steps (FTP / SFTP / SMB)
-
-FTP, SFTP, and SMB share an identical operation interface. Only the URL protocol and connection details differ.
-
-## URL Formats
-
-```
-ftp://user:password@host:port/basePath       (port 21)
-ftps://user:password@host:port/basePath      (port 21, TLS)
-sftp://user:password@host:port/basePath      (port 22)
-smb://user:password@host/sharename/basePath  (port 445)
-smb://domain\user:password@host/sharename/   (domain auth)
-```
-
-All support `<<variable>>` syntax in the URL. **For password-based auth, explicitly include credentials in the connection URL** using `<<systemId_username>>` and `<<systemId_password>>` placeholders (e.g. `sftp://<<mySystem_username>>:<<mySystem_password>>@host/path`). Authentication is never automatic. For SFTP private-key auth, use `credentials.privateKey`/`credentials.passphrase` instead of URL placeholders.
-
-SMB requires a share name as the first path segment. Additional segments become a base path prepended to all operation paths.
-
-## System Credentials
-
-Only store connection credentials: `host`, `port`, `username`, `password`, `privateKey`, `passphrase`, `domain` (SMB), `home_dir`. Paths and share names belong in the URL, not credentials.
-
-## Path Handling
-
-**All operation paths are relative to the base path in the URL.** Do NOT use absolute filesystem paths.
-
-If the system URL is `sftp://user:pass@host/home/sftptest/uploads`:
-
-- `report.csv` → `/home/sftptest/uploads/report.csv`
-- `subdir/data.csv` → `/home/sftptest/uploads/subdir/data.csv`
-
-Do NOT use `/home/sftptest/uploads/report.csv` — the base path is already set in the URL.
-
-If the URL has no base path (e.g., `sftp://user:pass@host`), operation paths are used as-is from the server root.
-
-## Body Format
-
-JSON string — single operation or array of operations (batch):
-
-### Single
-
-```json
-{ "operation": "get", "path": "data/report.csv" }
-```
-
-### Batch (sequential, same connection)
-
-```json
-[
-  { "operation": "mkdir", "path": "backup" },
-  { "operation": "get", "path": "data/report.csv" },
-  { "operation": "put", "path": "backup/report.csv", "content": "data here" }
-]
-```
-
-## Supported Operations
-
-| Operation | Required fields   | Returns                                                                 |
-| --------- | ----------------- | ----------------------------------------------------------------------- |
-| `list`    | `path`            | Array of `{ name, path, size, type, modifyTime, ... }`                  |
-| `get`     | `path`            | Auto-parsed file content (CSV→objects, JSON→parsed, etc.) or raw string |
-| `put`     | `path`, `content` | `{ success, message, size }`                                            |
-| `delete`  | `path`            | `{ success, message }`                                                  |
-| `rename`  | `path`, `newPath` | `{ success, message }`                                                  |
-| `mkdir`   | `path`            | `{ success, message }`                                                  |
-| `rmdir`   | `path`            | `{ success, message }`                                                  |
-| `exists`  | `path`            | `{ exists: boolean, path }`                                             |
-| `stat`    | `path`            | File metadata or `{ exists: false }`                                    |
-
-### `get` — Step File Keys
-
-Downloaded files are added to the runtime file store. See the file-handling skill for the full reference on file detection, aliasing, and the `RuntimeExecutionFile` shape.
-
-- `data` contains the auto-parsed content (CSV -> objects, JSON -> parsed, PDF -> structured, etc.)
-- the step result exposes `stepFileKeys`
-- later steps can reference downloaded files via `file::<stepId>.raw` (exact bytes), `file::<stepId>.base64` (base64 string), or `file::<stepId>.extracted` (parsed content)
-- for multi-file operations, use bracket notation: `file::<stepId>["report.csv"].raw`
-
-### `put` — Content Handling
-
-- `RawFileBytes` (from `file::<key>.raw`) → written as exact original bytes
-- String → written as-is
-- Buffer → written directly
-- Object/Array → JSON.stringified with 2-space indent
-
-## Connection Details
-
-### SFTP
-
-Uses `ssh2-sftp-client`. Supports password + private key auth (`credentials.privateKey`, `credentials.passphrase`).
-
-### FTP/FTPS
-
-Uses `basic-ftp`. FTPS uses TLS with `rejectUnauthorized: false`.
-
-### SMB
-
-Uses `@awo00/smb2`. Connection lifecycle: create client → authenticate (domain optional) → connect tree (share) → execute → disconnect/logoff/close. Entire operation wrapped in timeout race.
-
-All paths use forward slashes (`/`) — SMB library handles Windows conversion internally.
-
-Credentials must be injected into the URL using placeholders:
-
-```
-smb://<<systemId_username>>:<<systemId_password>>@fileserver.example.com/ShareName
-```
-
-For private systems via Secure Gateway, the host is `tunnelId.tunnel` (e.g., `my_tunnel.tunnel`).
-
-## Return Value
-
-Single operation → result directly. Multiple operations → array of results.

package/dist/skills/superglue/references/http-apis.md

@@ -1,248 +0,0 @@
-# HTTP APIs
-
-Covers REST APIs, GraphQL, webhooks, and any HTTP-based service.
-
-## Step Configuration
-
-```typescript
-{
-  type: "request",
-  systemId: "my_api",
-  url: "<<my_api_url>>/v1/users",              // RECOMMENDED: use system URL variable for base URL
-  method: "GET",                               // GET, POST, PUT, DELETE, PATCH
-  headers: { "Authorization": "Bearer <<my_api_access_token>>" }, // supports <<variables>> and JS expressions
-  queryParams: { "limit": "<<limit>>", "status": "active" }, // supports <<variables>> and JS expressions
-  body: '{"name": "<<(sourceData) => sourceData.currentItem.name>>"}', // supports <<variables>> and JS expressions
-  pagination: { ... }  // optional
-}
-```
-
-### URL Best Practices
-
-**Prefer system URL variables over hardcoded URLs:**
-
-```typescript
-// RECOMMENDED: Environment-agnostic (works with dev/prod switching)
-{ "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account" }
-
-// ACCEPTABLE: When URL differs significantly from system base URL
-{ "url": "https://api.stripe.com/v1/customers" }
-
-// AVOID: Hardcoding URLs that match the system's base URL
-{ "url": "https://mycompany.salesforce.com/services/data/v58.0/sobjects/Account" }
-```
-
-Using `<<systemId_url>>` enables the same tool to work across different environments (dev/prod) without modification. The URL resolves dynamically based on execution mode.
-
-## Authentication Patterns
-
-```javascript
-// Bearer token
-{ "Authorization": "Bearer <<systemId_access_token>>" }
-
-// API key in header
-{ "X-API-Key": "<<systemId_api_key>>" }
-
-// Basic Auth — auto-encoded to Base64, do NOT manually encode
-{ "Authorization": "Basic <<systemId_username>>:<<systemId_password>>" }
-
-// Runtime credentials from payload
-{ "Authorization": "Bearer <<(sourceData) => sourceData.user_access_token>>" }
-```
-
-Headers starting with `x-` are treated as custom headers.
-Modern APIs expect auth in headers, NOT query parameters, unless docs explicitly say otherwise.
-
-## Request Behavior
-
-- **GET, HEAD, DELETE, OPTIONS**: Body is always stripped
-- **POST, PUT, PATCH**: Body with leading `{` is JSON-parsed; empty body becomes undefined
-- Default headers: `Accept: */*`, Chrome-like `User-Agent`
-- HTTPS: `rejectUnauthorized: false` (accepts self-signed certs)
-
-### Response Handling
-
-All HTTP responses are read as raw bytes, then classified by `detectFileType` (byte-level detection). See the file-handling skill for the full detection logic and file type reference.
-
-- **Binary types** (PDF, Excel, DOCX, ZIP, GZIP): treated as files. Parsed content also becomes `data`. Later steps can reference the file via `file::<stepId>.raw/base64/extracted`.
-- **`application/octet-stream`**: always treated as a file, even if byte-level detection falls back to `RAW`.
-- **Large responses over 25 MB**: treated as files as a safety fallback so the runtime does not try to parse a huge unknown blob as RAW text/JSON.
-- **RAW + Content-Disposition attachment**: treated as a file.
-- **Smaller structured/text responses** (JSON, XML, CSV, YAML, HTML, plain text under the size limit): parsed inline to `data` only.
-
-This means an HTTP step downloading a PDF or large `.bin` file will expose `stepFileKeys` and populate `sourceData.__files__`, while an ordinary JSON API response will usually just populate `sourceData.stepId.data`.
-
-### Body Resolution Order
-
-For HTTP request steps, the runtime processes the body in this order:
-
-1. `<<>>` template variables are resolved via `replaceVariables`
-2. The resulting body string is JSON-parsed when possible
-3. `file::...` references are resolved on the parsed body object
-4. The HTTP strategy decides how to send the final body:
-   - raw bytes for `file::<key>.raw` when the **entire body** is a file ref
-   - base64 string content for `file::<key>.base64` when the API expects base64 inside JSON/XML
-   - `multipart/form-data` when the `Content-Type` header requests it and the body resolves to an object
-   - plain string body for strings
-   - `JSON.stringify(...)` for ordinary objects
-
-File references are resolved **only in the body**, not in headers, query params, or URLs.
-
-**Avoiding double-encoding:** The runtime stringifies objects at both step 1 (expression → template) and step 4 (object → fetch body). When the body contains nested JSON (e.g. LLM APIs), have `<<>>` expressions return a string via `JSON.stringify(...)`, or use a preceding transform step.
-
-### File Uploads
-
-#### Raw Body Uploads
-
-If the endpoint expects the file as the entire HTTP body, use:
-
-```typescript
-{
-  method: "PUT",
-  headers: { "Content-Type": "application/pdf" },
-  body: "file::document.raw"
-}
-```
-
-This sends the exact original file bytes as the request body.
-
-#### Base64-In-JSON/XML Uploads
-
-If the endpoint expects the file encoded as base64 inside a structured payload, use:
-
-```typescript
-{
-  method: "POST",
-  headers: { "Content-Type": "application/json" },
-  body: '{"fileBase64":"file::document.base64"}'
-}
-```
-
-This resolves the file bytes to a base64 string before the final `JSON.stringify(...)`.
-
-#### Automatic Multipart Uploads
-
-If the endpoint expects `multipart/form-data`, set the header and make the body a JSON object of form fields:
-
-```typescript
-{
-  method: "POST",
-  headers: { "Content-Type": "multipart/form-data" },
-  body: '{"file":"file::report.raw","description":"monthly export"}'
-}
-```
-
-The runtime will:
-
-- build a real `FormData` request
-- turn `file::...raw` values into file parts using the stored filename and content type
-- turn scalar values into text parts
-- JSON-stringify non-file objects/arrays into text parts
-- remove the user-supplied `Content-Type` header and let `fetch` set the multipart boundary automatically
-
-Restrictions:
-
-- the multipart body must resolve to an object of top-level form fields
-- nested file refs inside objects are not supported
-- do **not** manually include a multipart boundary in the header
-- use `.raw` for file parts, `.base64` for base64 text fields, and `.extracted` only for text fields that should contain parsed content
-
-## Error Detection
-
-### HTTP Status Errors
-
-Non-2xx responses throw with: method, URL, response body preview (1000 chars), masked config, retry count.
-
-### Smart Error Detection in 2xx Responses
-
-Even successful responses are scanned for error indicators:
-
-- `response.code` or `response.status` is 400-599 → throws
-- Keys matching `error`, `errors`, `error_message`, `failure_reason`, `failure`, `failed` (up to depth 2) with non-empty values → throws
-- Error message includes: _"To prevent this from happening, enable 'Continue on failure' in the step's Advanced Settings."_
-
-Bypass: `failureBehavior: "continue"` skips all error checking. Can be set in the tool playground.
-
-## Retry Logic
-
-### Connection/Server Errors
-
-- Default 1 retry (3 if keep-alive disabled), capped at server max
-- Only retries if response was fast (under quick-retry threshold)
-- Configurable delay between retries
-
-### Rate Limiting (429)
-
-- Respects `Retry-After` header (seconds or date)
-- Without header: exponential backoff `10^n * 1000ms + jitter` (max 1hr per wait)
-- Total wait capped at `MAX_RATE_LIMIT_WAIT_MS`
-- Separate from general retry count
-
-## Pagination
-
-Makes multiple requests in a loop, merging results. Only configure if you've verified the exact API pagination mechanism from docs.
-
-### Configuration
-
-```typescript
-pagination: {
-  type: "offsetBased" | "pageBased" | "cursorBased",
-  pageSize: "50",
-  cursorPath: "meta.next_cursor",   // cursorBased only — JSONPath to cursor
-  stopCondition: "(response, pageInfo) => !response.data.meta.next_cursor"
-}
-```
-
-### Variables (auto-injected)
-
-| Type        | Variable                     | Starts at | Increments by            |
-| ----------- | ---------------------------- | --------- | ------------------------ |
-| pageBased   | `<<page>>`                   | 1         | 1                        |
-| offsetBased | `<<offset>>`                 | 0         | pageSize                 |
-| cursorBased | `<<cursor>>`                 | null      | extracted via cursorPath |
-| all         | `<<limit>>` / `<<pageSize>>` | pageSize  | —                        |
-
-**CRITICAL**: The matching pagination variable MUST appear in the request (URL, queryParams, headers, or body). Throws if missing.
-
-### Stop Conditions
-
-JS expression evaluated in sandbox. Receives `(response, pageInfo)`:
-
-- `response.data` = parsed API response body
-- `response.headers` = response headers
-- `pageInfo = { page, offset, cursor, totalFetched }`
-- Return `true` to **STOP**
-
-Examples:
-
-```javascript
-"!response.data.meta.next_cursor"; // no next cursor
-"response.data.items.length === 0"; // empty page
-"response.data.hasMore === false"; // explicit flag
-"pageInfo.totalFetched >= 1000"; // item cap
-```
-
-### Safety Checks (only when stopCondition is set)
-
-1. **Identical pages 1 & 2** (both with data): Throws — pagination params aren't being applied
-2. **Both pages empty + stop didn't trigger**: Throws — broken stop condition
-3. **Duplicate consecutive response (after page 2)**: Auto-stops
-
-### Without stopCondition (fallback)
-
-- Array shorter than pageSize → stop
-- Duplicate response hash (against any previously seen page) → stop
-- Non-array response → stop after first request
-- Max 500 requests (vs 1000 max with a stopCondition)
-
-### Cursor Extraction
-
-`cursorPath` is a JSONPath expression (auto-prefixed with `$.` if needed). Extracted via `jsonpath-plus`. Null cursor → stop.
-
-**Special characters in cursorPath:** For property names containing `@` (e.g. OData's `@odata.nextLink`), use the property name directly as the cursorPath: `cursorPath: "@odata.nextLink"`. The engine handles `@`-prefixed paths automatically.
-
-## Output
-
-Paginated steps merge all pages into a **single** result object. The step result is always accessed via `sourceData.stepId.data` — it is NOT an array of per-page results. Do NOT call `.map()` on a paginated step result.
-
-Returns single object if one result, array if multiple (unwrapped from single-element array).