@superglue/cli 1.1.18 → 1.1.19

package/skills/superglue/references/file-servers.md

@@ -12,10 +12,14 @@ smb://user:password@host/sharename/basePath  (port 445)
 smb://domain\user:password@host/sharename/   (domain auth)
 ```
 
-All support `<<variable>>` syntax in the URL.
+All support `<<variable>>` syntax in the URL. **For password-based auth, explicitly include credentials in the connection URL** using `<<systemId_username>>` and `<<systemId_password>>` placeholders (e.g. `sftp://<<mySystem_username>>:<<mySystem_password>>@host/path`). Authentication is never automatic. For SFTP private-key auth, use `credentials.privateKey`/`credentials.passphrase` instead of URL placeholders.
 
 SMB requires a share name as the first path segment. Additional segments become a base path prepended to all operation paths.
 
+## System Credentials
+
+Only store connection credentials: `host`, `port`, `username`, `password`, `privateKey`, `passphrase`, `domain` (SMB), `home_dir`. Paths and share names belong in the URL, not credentials.
+
 ## Path Handling
 
 **All operation paths are relative to the base path in the URL.** Do NOT use absolute filesystem paths.
@@ -63,19 +67,21 @@ JSON string — single operation or array of operations (batch):
 | `exists`  | `path`            | `{ exists: boolean, path }`                                             |
 | `stat`    | `path`            | File metadata or `{ exists: false }`                                    |
 
-### `get` — Auto-Parsing
+### `get` — Step File Keys
 
-Downloaded files are all parsed automatically:
+Downloaded files are added to the runtime file store. See the file-handling skill for the full reference on file detection, aliasing, and the `RuntimeExecutionFile` shape.
 
-- CSV, JSON, XML, Excel, PDF, etc. → parsed to JS objects
-- Falls back to UTF-8 string
-- SMB additionally detects binary files (returns `{ _binary: true, encoding: "base64", data: "..." }`)
+- `data` contains the auto-parsed content (CSV -> objects, JSON -> parsed, PDF -> structured, etc.)
+- the step result exposes `stepFileKeys`
+- later steps can reference downloaded files via `file::<stepId>.raw` (exact bytes), `file::<stepId>.base64` (base64 string), or `file::<stepId>.extracted` (parsed content)
+- for multi-file operations, use bracket notation: `file::<stepId>["report.csv"].raw`
 
 ### `put` — Content Handling
 
+- `RawFileBytes` (from `file::<key>.raw`) → written as exact original bytes
 - String → written as-is
 - Buffer → written directly
-- Object/Array → `JSON.stringify` with 2-space indent
+- Object/Array → JSON.stringified with 2-space indent
 
 ## Connection Details
 
@@ -104,33 +110,3 @@ For private systems via Secure Gateway, the host is `tunnelId.tunnel` (e.g., `my
 ## Return Value
 
 Single operation → result directly. Multiple operations → array of results.
-
-## Common Patterns
-
-### Download and process
-
-```json
-{ "operation": "get", "path": "incoming/<<(sourceData) => sourceData.currentItem.filename>>" }
-```
-
-### Upload results
-
-```json
-{
-  "operation": "put",
-  "path": "output/results.json",
-  "content": "<<(sourceData) => JSON.stringify(sourceData.processData.data)>>"
-}
-```
-
-### List then loop
-
-Step 1 lists files, step 2 uses data selector to loop over them:
-
-```javascript
-// Step 1: list files
-// body: {"operation": "list", "path": "incoming"}
-
-// Step 2 dataSelector: (sourceData) => sourceData.listFiles.data.filter(f => f.type === "file")
-// body: {"operation": "get", "path": "<<(sourceData) => sourceData.currentItem.path>>"}
-```

package/skills/superglue/references/http-apis.md

@@ -0,0 +1,248 @@
+# HTTP APIs
+
+Covers REST APIs, GraphQL, webhooks, and any HTTP-based service.
+
+## Step Configuration
+
+```typescript
+{
+  type: "request",
+  systemId: "my_api",
+  url: "<<my_api_url>>/v1/users",              // RECOMMENDED: use system URL variable for base URL
+  method: "GET",                               // GET, POST, PUT, DELETE, PATCH
+  headers: { "Authorization": "Bearer <<my_api_access_token>>" }, // supports <<variables>> and JS expressions
+  queryParams: { "limit": "<<limit>>", "status": "active" }, // supports <<variables>> and JS expressions
+  body: '{"name": "<<(sourceData) => sourceData.currentItem.name>>"}', // supports <<variables>> and JS expressions
+  pagination: { ... }  // optional
+}
+```
+
+### URL Best Practices
+
+**Prefer system URL variables over hardcoded URLs:**
+
+```typescript
+// RECOMMENDED: Environment-agnostic (works with dev/prod switching)
+{ "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account" }
+
+// ACCEPTABLE: When URL differs significantly from system base URL
+{ "url": "https://api.stripe.com/v1/customers" }
+
+// AVOID: Hardcoding URLs that match the system's base URL
+{ "url": "https://mycompany.salesforce.com/services/data/v58.0/sobjects/Account" }
+```
+
+Using `<<systemId_url>>` enables the same tool to work across different environments (dev/prod) without modification. The URL resolves dynamically based on execution mode.
+
+## Authentication Patterns
+
+```javascript
+// Bearer token
+{ "Authorization": "Bearer <<systemId_access_token>>" }
+
+// API key in header
+{ "X-API-Key": "<<systemId_api_key>>" }
+
+// Basic Auth — auto-encoded to Base64, do NOT manually encode
+{ "Authorization": "Basic <<systemId_username>>:<<systemId_password>>" }
+
+// Runtime credentials from payload
+{ "Authorization": "Bearer <<(sourceData) => sourceData.user_access_token>>" }
+```
+
+Headers starting with `x-` are treated as custom headers.
+Modern APIs expect auth in headers, NOT query parameters, unless docs explicitly say otherwise.
+
+## Request Behavior
+
+- **GET, HEAD, DELETE, OPTIONS**: Body is always stripped
+- **POST, PUT, PATCH**: Body with leading `{` is JSON-parsed; empty body becomes undefined
+- Default headers: `Accept: */*`, Chrome-like `User-Agent`
+- HTTPS: `rejectUnauthorized: false` (accepts self-signed certs)
+
+### Response Handling
+
+All HTTP responses are read as raw bytes, then classified by `detectFileType` (byte-level detection). See the file-handling skill for the full detection logic and file type reference.
+
+- **Binary types** (PDF, Excel, DOCX, ZIP, GZIP): treated as files. Parsed content also becomes `data`. Later steps can reference the file via `file::<stepId>.raw/base64/extracted`.
+- **`application/octet-stream`**: always treated as a file, even if byte-level detection falls back to `RAW`.
+- **Large responses over 25 MB**: treated as files as a safety fallback so the runtime does not try to parse a huge unknown blob as RAW text/JSON.
+- **RAW + Content-Disposition attachment**: treated as a file.
+- **Smaller structured/text responses** (JSON, XML, CSV, YAML, HTML, plain text under the size limit): parsed inline to `data` only.
+
+This means an HTTP step downloading a PDF or large `.bin` file will expose `stepFileKeys` and populate `sourceData.__files__`, while an ordinary JSON API response will usually just populate `sourceData.stepId.data`.
+
+### Body Resolution Order
+
+For HTTP request steps, the runtime processes the body in this order:
+
+1. `<<>>` template variables are resolved via `replaceVariables`
+2. The resulting body string is JSON-parsed when possible
+3. `file::...` references are resolved on the parsed body object
+4. The HTTP strategy decides how to send the final body:
+   - raw bytes for `file::<key>.raw` when the **entire body** is a file ref
+   - base64 string content for `file::<key>.base64` when the API expects base64 inside JSON/XML
+   - `multipart/form-data` when the `Content-Type` header requests it and the body resolves to an object
+   - plain string body for strings
+   - `JSON.stringify(...)` for ordinary objects
+
+File references are resolved **only in the body**, not in headers, query params, or URLs.
+
+**Avoiding double-encoding:** The runtime stringifies objects at both step 1 (expression → template) and step 4 (object → fetch body). When the body contains nested JSON (e.g. LLM APIs), have `<<>>` expressions return a string via `JSON.stringify(...)`, or use a preceding transform step.
+
+### File Uploads
+
+#### Raw Body Uploads
+
+If the endpoint expects the file as the entire HTTP body, use:
+
+```typescript
+{
+  method: "PUT",
+  headers: { "Content-Type": "application/pdf" },
+  body: "file::document.raw"
+}
+```
+
+This sends the exact original file bytes as the request body.
+
+#### Base64-In-JSON/XML Uploads
+
+If the endpoint expects the file encoded as base64 inside a structured payload, use:
+
+```typescript
+{
+  method: "POST",
+  headers: { "Content-Type": "application/json" },
+  body: '{"fileBase64":"file::document.base64"}'
+}
+```
+
+This resolves the file bytes to a base64 string before the final `JSON.stringify(...)`.
+
+#### Automatic Multipart Uploads
+
+If the endpoint expects `multipart/form-data`, set the header and make the body a JSON object of form fields:
+
+```typescript
+{
+  method: "POST",
+  headers: { "Content-Type": "multipart/form-data" },
+  body: '{"file":"file::report.raw","description":"monthly export"}'
+}
+```
+
+The runtime will:
+
+- build a real `FormData` request
+- turn `file::...raw` values into file parts using the stored filename and content type
+- turn scalar values into text parts
+- JSON-stringify non-file objects/arrays into text parts
+- remove the user-supplied `Content-Type` header and let `fetch` set the multipart boundary automatically
+
+Restrictions:
+
+- the multipart body must resolve to an object of top-level form fields
+- nested file refs inside objects are not supported
+- do **not** manually include a multipart boundary in the header
+- use `.raw` for file parts, `.base64` for base64 text fields, and `.extracted` only for text fields that should contain parsed content
+
+## Error Detection
+
+### HTTP Status Errors
+
+Non-2xx responses throw with: method, URL, response body preview (1000 chars), masked config, retry count.
+
+### Smart Error Detection in 2xx Responses
+
+Even successful responses are scanned for error indicators:
+
+- `response.code` or `response.status` is 400-599 → throws
+- Keys matching `error`, `errors`, `error_message`, `failure_reason`, `failure`, `failed` (up to depth 2) with non-empty values → throws
+- Error message includes: _"To prevent this from happening, enable 'Continue on failure' in the step's Advanced Settings."_
+
+Bypass: `failureBehavior: "continue"` skips all error checking. Can be set in the tool playground.
+
+## Retry Logic
+
+### Connection/Server Errors
+
+- Default 1 retry (3 if keep-alive disabled), capped at server max
+- Only retries if response was fast (under quick-retry threshold)
+- Configurable delay between retries
+
+### Rate Limiting (429)
+
+- Respects `Retry-After` header (seconds or date)
+- Without header: exponential backoff `10^n * 1000ms + jitter` (max 1hr per wait)
+- Total wait capped at `MAX_RATE_LIMIT_WAIT_MS`
+- Separate from general retry count
+
+## Pagination
+
+Makes multiple requests in a loop, merging results. Only configure if you've verified the exact API pagination mechanism from docs.
+
+### Configuration
+
+```typescript
+pagination: {
+  type: "offsetBased" | "pageBased" | "cursorBased",
+  pageSize: "50",
+  cursorPath: "meta.next_cursor",   // cursorBased only — JSONPath to cursor
+  stopCondition: "(response, pageInfo) => !response.data.meta.next_cursor"
+}
+```
+
+### Variables (auto-injected)
+
+| Type        | Variable                     | Starts at | Increments by            |
+| ----------- | ---------------------------- | --------- | ------------------------ |
+| pageBased   | `<<page>>`                   | 1         | 1                        |
+| offsetBased | `<<offset>>`                 | 0         | pageSize                 |
+| cursorBased | `<<cursor>>`                 | null      | extracted via cursorPath |
+| all         | `<<limit>>` / `<<pageSize>>` | pageSize  | —                        |
+
+**CRITICAL**: The matching pagination variable MUST appear in the request (URL, queryParams, headers, or body). Throws if missing.
+
+### Stop Conditions
+
+JS expression evaluated in sandbox. Receives `(response, pageInfo)`:
+
+- `response.data` = parsed API response body
+- `response.headers` = response headers
+- `pageInfo = { page, offset, cursor, totalFetched }`
+- Return `true` to **STOP**
+
+Examples:
+
+```javascript
+"!response.data.meta.next_cursor"; // no next cursor
+"response.data.items.length === 0"; // empty page
+"response.data.hasMore === false"; // explicit flag
+"pageInfo.totalFetched >= 1000"; // item cap
+```
+
+### Safety Checks (only when stopCondition is set)
+
+1. **Identical pages 1 & 2** (both with data): Throws — pagination params aren't being applied
+2. **Both pages empty + stop didn't trigger**: Throws — broken stop condition
+3. **Duplicate consecutive response (after page 2)**: Auto-stops
+
+### Without stopCondition (fallback)
+
+- Array shorter than pageSize → stop
+- Duplicate response hash (against any previously seen page) → stop
+- Non-array response → stop after first request
+- Max 500 requests (vs 1000 max with a stopCondition)
+
+### Cursor Extraction
+
+`cursorPath` is a JSONPath expression (auto-prefixed with `$.` if needed). Extracted via `jsonpath-plus`. Null cursor → stop.
+
+**Special characters in cursorPath:** For property names containing `@` (e.g. OData's `@odata.nextLink`), use the property name directly as the cursorPath: `cursorPath: "@odata.nextLink"`. The engine handles `@`-prefixed paths automatically.
+
+## Output
+
+Paginated steps merge all pages into a **single** result object. The step result is always accessed via `sourceData.stepId.data` — it is NOT an array of per-page results. Do NOT call `.map()` on a paginated step result.
+
+Returns single object if one result, array if multiple (unwrapped from single-element array).

package/skills/superglue/references/redis.md

@@ -7,7 +7,6 @@
   type: "request",
   systemId: "my_redis",
   url: "redis://<<my_redis_username>>:<<my_redis_password>>@<<my_redis_host>>:<<my_redis_port>>/<<my_redis_database>>",
-  method: "POST",            // ignored for Redis but required by schema
   body: '{"command": "GET", "args": ["<<keyName>>"]}'
 }
 ```
@@ -23,7 +22,7 @@ rediss://user:password@host:port/database   (TLS)
 
 - `redis://` for standard connections
 - `rediss://` for TLS-encrypted connections (e.g. Redis Cloud, AWS ElastiCache with encryption)
-- `/database` is the database number (0-15), defaults to 0 if omitted
+- `/database` is the database number (defaults to 0 if omitted; max depends on the server's `databases` config, commonly 16)
 
 All credential variables are resolved before connecting. Trailing slashes are cleaned.
 
@@ -120,7 +119,7 @@ Multiple commands:
 ## Connection Management
 
 - A fresh connection is created per execution and closed after completion
-- TLS auto-configured for `rediss://` URLs (with `rejectUnauthorized: false`)
+- TLS auto-configured for `rediss://` URLs (currently uses `rejectUnauthorized: false`)
 - 5-second connection timeout, 30-second command timeout
 - Array commands (pipeline) run on a single connection for efficiency
 
@@ -133,7 +132,7 @@ Default retries from server config. On final failure, error includes command and
 Returns the raw Redis response:
 
 - `GET` / `HGET`: string or null
-- `HGETALL`: object with field-value pairs (flattened array from Redis, parsed by ioredis)
+- `HGETALL`: flat array of alternating field-value pairs (e.g., `["name", "Alice", "email", "alice@example.com"]`)
 - `LRANGE` / `SMEMBERS` / `KEYS`: array of strings
 - `SET` / `DEL` / `EXPIRE`: "OK" or integer
 - `INCR` / `LLEN` / `SADD`: integer
@@ -186,5 +185,5 @@ Use a data selector returning an array to execute a command per item:
 
 - All args must be strings (Redis protocol is text-based) - numbers are auto-coerced
 - Use `SCAN` instead of `KEYS` in production for large keyspaces
-- `HGETALL` returns an empty object `{}` for non-existent keys
+- `HGETALL` returns an empty array `[]` for non-existent keys
 - Use array body for multi-command pipelines instead of separate steps