@superglue/cli 1.1.21 → 1.1.23

package/dist/skills/superglue/references/http-apis.md

@@ -1,248 +0,0 @@
-# HTTP APIs
-
-Covers REST APIs, GraphQL, webhooks, and any HTTP-based service.
-
-## Step Configuration
-
-```typescript
-{
-  type: "request",
-  systemId: "my_api",
-  url: "<<my_api_url>>/v1/users",              // RECOMMENDED: use system URL variable for base URL
-  method: "GET",                               // GET, POST, PUT, DELETE, PATCH
-  headers: { "Authorization": "Bearer <<my_api_access_token>>" }, // supports <<variables>> and JS expressions
-  queryParams: { "limit": "<<limit>>", "status": "active" }, // supports <<variables>> and JS expressions
-  body: '{"name": "<<(sourceData) => sourceData.currentItem.name>>"}', // supports <<variables>> and JS expressions
-  pagination: { ... }  // optional
-}
-```
-
-### URL Best Practices
-
-**Prefer system URL variables over hardcoded URLs:**
-
-```typescript
-// RECOMMENDED: Environment-agnostic (works with dev/prod switching)
-{ "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account" }
-
-// ACCEPTABLE: When URL differs significantly from system base URL
-{ "url": "https://api.stripe.com/v1/customers" }
-
-// AVOID: Hardcoding URLs that match the system's base URL
-{ "url": "https://mycompany.salesforce.com/services/data/v58.0/sobjects/Account" }
-```
-
-Using `<<systemId_url>>` enables the same tool to work across different environments (dev/prod) without modification. The URL resolves dynamically based on execution mode.
-
-## Authentication Patterns
-
-```javascript
-// Bearer token
-{ "Authorization": "Bearer <<systemId_access_token>>" }
-
-// API key in header
-{ "X-API-Key": "<<systemId_api_key>>" }
-
-// Basic Auth — auto-encoded to Base64, do NOT manually encode
-{ "Authorization": "Basic <<systemId_username>>:<<systemId_password>>" }
-
-// Runtime credentials from payload
-{ "Authorization": "Bearer <<(sourceData) => sourceData.user_access_token>>" }
-```
-
-Headers starting with `x-` are treated as custom headers.
-Modern APIs expect auth in headers, NOT query parameters, unless docs explicitly say otherwise.
-
-## Request Behavior
-
-- **GET, HEAD, DELETE, OPTIONS**: Body is always stripped
-- **POST, PUT, PATCH**: Body with leading `{` is JSON-parsed; empty body becomes undefined
-- Default headers: `Accept: */*`, Chrome-like `User-Agent`
-- HTTPS: `rejectUnauthorized: false` (accepts self-signed certs)
-
-### Response Handling
-
-All HTTP responses are read as raw bytes, then classified by `detectFileType` (byte-level detection). See the file-handling skill for the full detection logic and file type reference.
-
-- **Binary types** (PDF, Excel, DOCX, ZIP, GZIP): treated as files. Parsed content also becomes `data`. Later steps can reference the file via `file::<stepId>.raw/base64/extracted`.
-- **`application/octet-stream`**: always treated as a file, even if byte-level detection falls back to `RAW`.
-- **Large responses over 25 MB**: treated as files as a safety fallback so the runtime does not try to parse a huge unknown blob as RAW text/JSON.
-- **RAW + Content-Disposition attachment**: treated as a file.
-- **Smaller structured/text responses** (JSON, XML, CSV, YAML, HTML, plain text under the size limit): parsed inline to `data` only.
-
-This means an HTTP step downloading a PDF or large `.bin` file will expose `stepFileKeys` and populate `sourceData.__files__`, while an ordinary JSON API response will usually just populate `sourceData.stepId.data`.
-
-### Body Resolution Order
-
-For HTTP request steps, the runtime processes the body in this order:
-
-1. `<<>>` template variables are resolved via `replaceVariables`
-2. The resulting body string is JSON-parsed when possible
-3. `file::...` references are resolved on the parsed body object
-4. The HTTP strategy decides how to send the final body:
-   - raw bytes for `file::<key>.raw` when the **entire body** is a file ref
-   - base64 string content for `file::<key>.base64` when the API expects base64 inside JSON/XML
-   - `multipart/form-data` when the `Content-Type` header requests it and the body resolves to an object
-   - plain string body for strings
-   - `JSON.stringify(...)` for ordinary objects
-
-File references are resolved **only in the body**, not in headers, query params, or URLs.
-
-**Avoiding double-encoding:** The runtime stringifies objects at both step 1 (expression → template) and step 4 (object → fetch body). When the body contains nested JSON (e.g. LLM APIs), have `<<>>` expressions return a string via `JSON.stringify(...)`, or use a preceding transform step.
-
-### File Uploads
-
-#### Raw Body Uploads
-
-If the endpoint expects the file as the entire HTTP body, use:
-
-```typescript
-{
-  method: "PUT",
-  headers: { "Content-Type": "application/pdf" },
-  body: "file::document.raw"
-}
-```
-
-This sends the exact original file bytes as the request body.
-
-#### Base64-In-JSON/XML Uploads
-
-If the endpoint expects the file encoded as base64 inside a structured payload, use:
-
-```typescript
-{
-  method: "POST",
-  headers: { "Content-Type": "application/json" },
-  body: '{"fileBase64":"file::document.base64"}'
-}
-```
-
-This resolves the file bytes to a base64 string before the final `JSON.stringify(...)`.
-
-#### Automatic Multipart Uploads
-
-If the endpoint expects `multipart/form-data`, set the header and make the body a JSON object of form fields:
-
-```typescript
-{
-  method: "POST",
-  headers: { "Content-Type": "multipart/form-data" },
-  body: '{"file":"file::report.raw","description":"monthly export"}'
-}
-```
-
-The runtime will:
-
-- build a real `FormData` request
-- turn `file::...raw` values into file parts using the stored filename and content type
-- turn scalar values into text parts
-- JSON-stringify non-file objects/arrays into text parts
-- remove the user-supplied `Content-Type` header and let `fetch` set the multipart boundary automatically
-
-Restrictions:
-
-- the multipart body must resolve to an object of top-level form fields
-- nested file refs inside objects are not supported
-- do **not** manually include a multipart boundary in the header
-- use `.raw` for file parts, `.base64` for base64 text fields, and `.extracted` only for text fields that should contain parsed content
-
-## Error Detection
-
-### HTTP Status Errors
-
-Non-2xx responses throw with: method, URL, response body preview (1000 chars), masked config, retry count.
-
-### Smart Error Detection in 2xx Responses
-
-Even successful responses are scanned for error indicators:
-
-- `response.code` or `response.status` is 400-599 → throws
-- Keys matching `error`, `errors`, `error_message`, `failure_reason`, `failure`, `failed` (up to depth 2) with non-empty values → throws
-- Error message includes: _"To prevent this from happening, enable 'Continue on failure' in the step's Advanced Settings."_
-
-Bypass: `failureBehavior: "continue"` skips all error checking. Can be set in the tool playground.
-
-## Retry Logic
-
-### Connection/Server Errors
-
-- Default 1 retry (3 if keep-alive disabled), capped at server max
-- Only retries if response was fast (under quick-retry threshold)
-- Configurable delay between retries
-
-### Rate Limiting (429)
-
-- Respects `Retry-After` header (seconds or date)
-- Without header: exponential backoff `10^n * 1000ms + jitter` (max 1hr per wait)
-- Total wait capped at `MAX_RATE_LIMIT_WAIT_MS`
-- Separate from general retry count
-
-## Pagination
-
-Makes multiple requests in a loop, merging results. Only configure if you've verified the exact API pagination mechanism from docs.
-
-### Configuration
-
-```typescript
-pagination: {
-  type: "offsetBased" | "pageBased" | "cursorBased",
-  pageSize: "50",
-  cursorPath: "meta.next_cursor",   // cursorBased only — JSONPath to cursor
-  stopCondition: "(response, pageInfo) => !response.data.meta.next_cursor"
-}
-```
-
-### Variables (auto-injected)
-
-| Type        | Variable                     | Starts at | Increments by            |
-| ----------- | ---------------------------- | --------- | ------------------------ |
-| pageBased   | `<<page>>`                   | 1         | 1                        |
-| offsetBased | `<<offset>>`                 | 0         | pageSize                 |
-| cursorBased | `<<cursor>>`                 | null      | extracted via cursorPath |
-| all         | `<<limit>>` / `<<pageSize>>` | pageSize  | —                        |
-
-**CRITICAL**: The matching pagination variable MUST appear in the request (URL, queryParams, headers, or body). Throws if missing.
-
-### Stop Conditions
-
-JS expression evaluated in sandbox. Receives `(response, pageInfo)`:
-
-- `response.data` = parsed API response body
-- `response.headers` = response headers
-- `pageInfo = { page, offset, cursor, totalFetched }`
-- Return `true` to **STOP**
-
-Examples:
-
-```javascript
-"!response.data.meta.next_cursor"; // no next cursor
-"response.data.items.length === 0"; // empty page
-"response.data.hasMore === false"; // explicit flag
-"pageInfo.totalFetched >= 1000"; // item cap
-```
-
-### Safety Checks (only when stopCondition is set)
-
-1. **Identical pages 1 & 2** (both with data): Throws — pagination params aren't being applied
-2. **Both pages empty + stop didn't trigger**: Throws — broken stop condition
-3. **Duplicate consecutive response (after page 2)**: Auto-stops
-
-### Without stopCondition (fallback)
-
-- Array shorter than pageSize → stop
-- Duplicate response hash (against any previously seen page) → stop
-- Non-array response → stop after first request
-- Max 500 requests (vs 1000 max with a stopCondition)
-
-### Cursor Extraction
-
-`cursorPath` is a JSONPath expression (auto-prefixed with `$.` if needed). Extracted via `jsonpath-plus`. Null cursor → stop.
-
-**Special characters in cursorPath:** For property names containing `@` (e.g. OData's `@odata.nextLink`), use the property name directly as the cursorPath: `cursorPath: "@odata.nextLink"`. The engine handles `@`-prefixed paths automatically.
-
-## Output
-
-Paginated steps merge all pages into a **single** result object. The step result is always accessed via `sourceData.stepId.data` — it is NOT an array of per-page results. Do NOT call `.map()` on a paginated step result.
-
-Returns single object if one result, array if multiple (unwrapped from single-element array).

package/skills/superglue/references/data-handling.md

@@ -1,303 +0,0 @@
-# Data Handling: Variables, Selectors & Transforms
-
-Most tool building and editing failures trace back to incorrect variable syntax, wrong data selector returns, or misunderstanding of the tool result envelope.
-
-## Variable Replacement (`<<>>` Syntax)
-
-The `<<expression>>` syntax injects dynamic values into step config fields (URL, headers, body, queryParams).
-
-### Simple Variables (top-level keys only)
-
-```
-<<userId>>           payload keys
-<<currentItem>>      whole current loop item
-<<page>>             pagination variable
-<<systemId_api_key>> system credential
-<<systemId_url>>     system base URL
-<<sg_auth_email>>    email of the authenticated user
-```
-
-**CRITICAL**: Simple `<<varName>>` only works for top-level keys. NO dots, NO nesting.
-
-- VALID: `<<userId>>`, `<<currentItem>>`, `<<page>>`, `<<stripe_api_key>>`, `<<stripe_url>>`, `<<sg_auth_email>>`
-- INVALID: `<<currentItem.id>>`, `<<sourceData.userId>>`, `<<user.name>>` — these FAIL at runtime
-
-### Arrow Function Expressions (for everything else)
-
-```
-<<(sourceData) => sourceData.currentItem.player_id>>
-<<(sourceData) => sourceData.getUsers.data.map(u => u.id)>>
-<<(sourceData) => JSON.stringify({ ids: sourceData.fetchUsers.data.map(u => u.id) })>>
-```
-
-### Resolution Order
-
-1. Direct lookup: check if expression is a top-level key in the merged variable object
-2. Arrow function: execute in Deno subprocess sandbox with full data as `sourceData`
-3. Failure: throws error listing available keys
-
-### Variable Sources
-
-All variables come from a single merged object:
-
-```javascript
-{
-  ...originalPayload,       // user-provided input
-  ...previousStepResults,   // keyed by stepId
-  ...systemCredentials,     // namespaced: systemId_credKey
-  ...systemUrls,            // namespaced: systemId_url
-  sg_auth_email,            // email of authenticated user (context variable)
-  currentItem,              // current loop item (if in a loop)
-  page, offset, cursor,     // pagination variables
-  limit, pageSize           // pagination size
-}
-```
-
-## Context Variables
-
-Context variables provide information about the execution environment:
-
-- `<<sg_auth_email>>` — Email address of the authenticated user who triggered the tool execution
-
-**Use case**: Personalize tool outputs based on who is running the tool. For example:
-
-- Different permissions in Tableau based on user
-- User-specific account lookups
-- Audit logging with user identity
-
-**Important**: `sg_auth_email` is NOT available in scheduled executions (cron jobs). If a tool references `<<sg_auth_email>>` and runs via scheduler, it will fail with a clear error message.
-
-### Return Type Handling
-
-- Objects/arrays returned from expressions → JSON.stringified automatically
-- Resolved values of `"undefined"` or `"null"` → stripped from headers and queryParams
-
-### Special: Basic Auth
-
-`"Authorization": "Basic <<user>>:<<pass>>"` is auto-converted to Base64. Do NOT manually encode.
-
-## Credential Resolution
-
-Credentials are available via `<<>>` syntax. Two sources, payload takes precedence:
-
-1. **System credentials**: Stored on System, namespaced as `<<systemId_credKey>>`
-
-   ```javascript
-   // System id="stripe", credentials={ api_key: "sk_..." }
-   // Available as: <<stripe_api_key>>
-   ```
-
-2. **Payload credentials**: Passed at runtime, accessed directly
-   ```javascript
-   // Payload: { user_access_token: "abc" }
-   // Available as: <<user_access_token>>
-   // Or: <<(sourceData) => sourceData.user_access_token>>
-   ```
-
-For OAuth systems, use `<<systemId_access_token>>` — the token value is auto-refreshed during tool execution, but the header itself must be present in the step config.
-
-## System URL Resolution
-
-System URLs are available via `<<systemId_url>>` syntax. This enables environment-agnostic tools:
-
-```javascript
-// System id="salesforce", url="https://mycompany.salesforce.com"
-// Available as: <<salesforce_url>>
-
-// Step config:
-{
-  "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account",
-  "headers": { "Authorization": "Bearer <<salesforce_access_token>>" }
-}
-```
-
-**Why use system URL variables?**
-
-- Same tool works across dev/prod environments without modification
-- When switching execution mode, the system URL resolves to the appropriate environment
-- Avoids hardcoding base URLs that differ between environments
-
-## Data Selectors
-
-Every step has a `dataSelector` — a JS function that controls execution mode and what `currentItem` is.
-
-### Returns OBJECT → Single Execution
-
-Step runs once. The object becomes `currentItem`.
-
-```javascript
-(sourceData) => ({ userId: sourceData.userId })
-(sourceData) => ({})   // run once with no specific input
-```
-
-Result stored as:
-
-```javascript
-sourceData.stepId = { currentItem: <object>, data: <response>, success: true }
-```
-
-### Returns ARRAY → Loop Execution
-
-Step runs once per array element. Each element becomes `currentItem`.
-
-```javascript
-(sourceData) => sourceData.getContacts.data.filter(c => c.active)
-(sourceData) => sourceData.userIds   // if userIds is an array from payload
-```
-
-Result stored as:
-
-```javascript
-sourceData.stepId = [
-  { currentItem: <item1>, data: <response1>, success: true },
-  { currentItem: <item2>, data: <response2>, success: true },
-  ...
-]
-```
-
-Empty arrays are valid — step just skips execution.
-
-### Accessing currentItem in Step Config
-
-```
-<<currentItem>>                                           // whole value
-<<(sourceData) => sourceData.currentItem.id>>             // property
-<<(sourceData) => sourceData.currentItem.name.toUpperCase()>>  // with transform
-```
-
-## Result Envelope
-
-**Every** step result (request or transform) is wrapped identically:
-
-### From Object Selector
-
-```javascript
-sourceData.stepId = {
-  currentItem: <selectorOutput>,
-  data: <apiResponse or transformResult>,
-  success: true
-}
-
-// Access:
-sourceData.stepId.data           // the response
-sourceData.stepId.currentItem    // what was passed in
-```
-
-### From Array Selector
-
-```javascript
-sourceData.stepId = [
-  { currentItem: <item1>, data: <response1>, success: true },
-  { currentItem: <item2>, data: <response2>, success: false, error: "..." },
-  ...
-]
-
-// Access:
-sourceData.stepId.map(item => item.data)                     // all responses
-sourceData.stepId.flatMap(item => item.data.results)          // nested arrays
-sourceData.stepId.filter(item => item.success).map(i => i.data) // only successes
-```
-
-### From Paginated Step
-
-Paginated steps merge all pages server-side into a **single** result. The result is always a single envelope, NOT an array — even though multiple pages were fetched.
-
-```javascript
-sourceData.stepId = {
-  data: <merged response from all pages>,
-  success: true
-}
-
-// Access:
-sourceData.stepId.data              // the merged response
-sourceData.stepId.data.items        // if the API wraps results in .items
-
-// WRONG — do NOT treat paginated results as an array of envelopes:
-sourceData.stepId.map(...)          // FAILS — stepId is an object, not an array
-```
-
-## sourceData Structure
-
-`sourceData` is the cumulative state available to data selectors, variable expressions, and transforms:
-
-```javascript
-sourceData = {
-  // Original payload fields at ROOT level (NOT under .payload)
-  userId: "abc",
-  date: "2024-01-15",
-  companies: ["acme", "globex"],
-
-  // Previous step results, keyed by step ID
-  getUsers: { currentItem: {}, data: { users: [...] }, success: true },
-  fetchDetails: [
-    { currentItem: { id: 1 }, data: { name: "Alice" }, success: true },
-    { currentItem: { id: 2 }, data: { name: "Bob" }, success: true },
-  ],
-
-  // Current item (only within a loop step's config)
-  currentItem: { id: 1 },
-
-  // Runtime file map — see file-handling skill for full reference
-  __files__: { ... }
-}
-```
-
-Treat `sourceData.__files__` as read-only. See the file-handling skill for file detection, the `RuntimeExecutionFile` shape, aliasing rules, and how to access files in transforms.
-
-## Three Transformation Points
-
-| Point                            | When it runs             | Purpose                              |
-| -------------------------------- | ------------------------ | ------------------------------------ |
-| `dataSelector`                   | Before step executes     | Controls input + single vs loop mode |
-| Transform step (`transformCode`) | As the step itself       | Reshape data between request steps   |
-| `outputTransform`                | After all steps complete | Shape final tool output              |
-
-## Transform Steps
-
-Transform steps use `type: "transform"` with a `transformCode` function. They do NOT have: systemId, url, method, headers, body, queryParams, or pagination.
-
-Results are wrapped in the **same envelope** as request steps:
-
-```javascript
-sourceData.formatForInsert = { currentItem: <dataSelector output>, data: <transformCode result>, success: true }
-```
-
-If the step has a dataSelector returning an array, the transform runs once per item. See tool-building guidelines for when to use transform steps vs dataSelectors vs outputTransform.
-
-### Returning Files From a Transform
-
-Transform steps can produce files by returning a reserved `__files__` key in the result. See the file-handling skill for the full shape, aliasing rules, and access patterns.
-
-## Output Transform
-
-Final transformation shaping the tool's output. Runs after all steps complete. Can access cumulative step result data and payloads.
-
-### Requirements
-
-- Function signature: `(sourceData) => { ... }`
-- **Must have a return statement**
-- Async/await IS supported (same Deno sandbox as transform steps)
-- NEVER include newlines or tabs in the code string
-
-## JS Code Constraints (all transform points)
-
-Runs in a Deno subprocess sandbox:
-
-- 8192 MB memory limit, 10 min timeout (transforms/selectors), 3s timeout (pagination stops)
-- Async/await IS supported in transforms
-- No filesystem access — `--deny-read`, `--deny-write`
-- No subprocess spawning — `--deny-run`
-- Network access IS available (`--allow-net`) — transforms can make HTTP requests
-- Environment variables are sandboxed — only safe vars (PATH, HOME, DENO_DIR) are visible, no server secrets
-- JSON-serializable I/O only
-- Must return a value (undefined returns null)
-
-### Serialization Safety
-
-All JS code is stored as a JSON string, then MessagePack-serialized to the Deno subprocess, then `eval()`'d. Patterns that break across this pipeline:
-
-- **Regex literals with `/`** — `/https?:\/\/([^/]+)/` corrupts. Use `new URL(str).hostname` or `str.split("//")[1].split("/")[0]` instead.
-- **Backslash-heavy patterns** — `\n`, `\t`, `\\` in regex/strings get misinterpreted. Prefer string methods (`.split()`, `.indexOf()`, `.includes()`) over regex.
-- **Literal newlines** — code string must be single-line. Use `;` between statements.
-
-**Rule of thumb**: prefer string methods over regex. When regex is unavoidable, use `new RegExp("pattern")` constructor over `/pattern/` literals.