@superglue/cli 1.1.18 → 1.1.20

package/dist/skills/superglue/references/data-handling.md

@@ -0,0 +1,303 @@
+# Data Handling: Variables, Selectors & Transforms
+
+Most tool building and editing failures trace back to incorrect variable syntax, wrong data selector returns, or misunderstanding of the tool result envelope.
+
+## Variable Replacement (`<<>>` Syntax)
+
+The `<<expression>>` syntax injects dynamic values into step config fields (URL, headers, body, queryParams).
+
+### Simple Variables (top-level keys only)
+
+```
+<<userId>>           payload keys
+<<currentItem>>      whole current loop item
+<<page>>             pagination variable
+<<systemId_api_key>> system credential
+<<systemId_url>>     system base URL
+<<sg_auth_email>>    email of the authenticated user
+```
+
+**CRITICAL**: Simple `<<varName>>` only works for top-level keys. NO dots, NO nesting.
+
+- VALID: `<<userId>>`, `<<currentItem>>`, `<<page>>`, `<<stripe_api_key>>`, `<<stripe_url>>`, `<<sg_auth_email>>`
+- INVALID: `<<currentItem.id>>`, `<<sourceData.userId>>`, `<<user.name>>` — these FAIL at runtime
+
+### Arrow Function Expressions (for everything else)
+
+```
+<<(sourceData) => sourceData.currentItem.player_id>>
+<<(sourceData) => sourceData.getUsers.data.map(u => u.id)>>
+<<(sourceData) => JSON.stringify({ ids: sourceData.fetchUsers.data.map(u => u.id) })>>
+```
+
+### Resolution Order
+
+1. Direct lookup: check if expression is a top-level key in the merged variable object
+2. Arrow function: execute in Deno subprocess sandbox with full data as `sourceData`
+3. Failure: throws error listing available keys
+
+### Variable Sources
+
+All variables come from a single merged object:
+
+```javascript
+{
+  ...originalPayload,       // user-provided input
+  ...previousStepResults,   // keyed by stepId
+  ...systemCredentials,     // namespaced: systemId_credKey
+  ...systemUrls,            // namespaced: systemId_url
+  sg_auth_email,            // email of authenticated user (context variable)
+  currentItem,              // current loop item (if in a loop)
+  page, offset, cursor,     // pagination variables
+  limit, pageSize           // pagination size
+}
+```
+
+## Context Variables
+
+Context variables provide information about the execution environment:
+
+- `<<sg_auth_email>>` — Email address of the authenticated user who triggered the tool execution
+
+**Use case**: Personalize tool outputs based on who is running the tool. For example:
+
+- Different permissions in Tableau based on user
+- User-specific account lookups
+- Audit logging with user identity
+
+**Important**: `sg_auth_email` is NOT available in scheduled executions (cron jobs). If a tool references `<<sg_auth_email>>` and runs via scheduler, it will fail with a clear error message.
+
+### Return Type Handling
+
+- Objects/arrays returned from expressions → JSON.stringified automatically
+- Resolved values of `"undefined"` or `"null"` → stripped from headers and queryParams
+
+### Special: Basic Auth
+
+`"Authorization": "Basic <<user>>:<<pass>>"` is auto-converted to Base64. Do NOT manually encode.
+
+## Credential Resolution
+
+Credentials are available via `<<>>` syntax. Two sources, payload takes precedence:
+
+1. **System credentials**: Stored on System, namespaced as `<<systemId_credKey>>`
+
+   ```javascript
+   // System id="stripe", credentials={ api_key: "sk_..." }
+   // Available as: <<stripe_api_key>>
+   ```
+
+2. **Payload credentials**: Passed at runtime, accessed directly
+   ```javascript
+   // Payload: { user_access_token: "abc" }
+   // Available as: <<user_access_token>>
+   // Or: <<(sourceData) => sourceData.user_access_token>>
+   ```
+
+For OAuth systems, use `<<systemId_access_token>>` — the token value is auto-refreshed during tool execution, but the header itself must be present in the step config.
+
+## System URL Resolution
+
+System URLs are available via `<<systemId_url>>` syntax. This enables environment-agnostic tools:
+
+```javascript
+// System id="salesforce", url="https://mycompany.salesforce.com"
+// Available as: <<salesforce_url>>
+
+// Step config:
+{
+  "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account",
+  "headers": { "Authorization": "Bearer <<salesforce_access_token>>" }
+}
+```
+
+**Why use system URL variables?**
+
+- Same tool works across dev/prod environments without modification
+- When switching execution mode, the system URL resolves to the appropriate environment
+- Avoids hardcoding base URLs that differ between environments
+
+## Data Selectors
+
+Every step has a `dataSelector` — a JS function that controls execution mode and what `currentItem` is.
+
+### Returns OBJECT → Single Execution
+
+Step runs once. The object becomes `currentItem`.
+
+```javascript
+(sourceData) => ({ userId: sourceData.userId })
+(sourceData) => ({})   // run once with no specific input
+```
+
+Result stored as:
+
+```javascript
+sourceData.stepId = { currentItem: <object>, data: <response>, success: true }
+```
+
+### Returns ARRAY → Loop Execution
+
+Step runs once per array element. Each element becomes `currentItem`.
+
+```javascript
+(sourceData) => sourceData.getContacts.data.filter(c => c.active)
+(sourceData) => sourceData.userIds   // if userIds is an array from payload
+```
+
+Result stored as:
+
+```javascript
+sourceData.stepId = [
+  { currentItem: <item1>, data: <response1>, success: true },
+  { currentItem: <item2>, data: <response2>, success: true },
+  ...
+]
+```
+
+Empty arrays are valid — step just skips execution.
+
+### Accessing currentItem in Step Config
+
+```
+<<currentItem>>                                           // whole value
+<<(sourceData) => sourceData.currentItem.id>>             // property
+<<(sourceData) => sourceData.currentItem.name.toUpperCase()>>  // with transform
+```
+
+## Result Envelope
+
+**Every** step result (request or transform) is wrapped identically:
+
+### From Object Selector
+
+```javascript
+sourceData.stepId = {
+  currentItem: <selectorOutput>,
+  data: <apiResponse or transformResult>,
+  success: true
+}
+
+// Access:
+sourceData.stepId.data           // the response
+sourceData.stepId.currentItem    // what was passed in
+```
+
+### From Array Selector
+
+```javascript
+sourceData.stepId = [
+  { currentItem: <item1>, data: <response1>, success: true },
+  { currentItem: <item2>, data: <response2>, success: false, error: "..." },
+  ...
+]
+
+// Access:
+sourceData.stepId.map(item => item.data)                     // all responses
+sourceData.stepId.flatMap(item => item.data.results)          // nested arrays
+sourceData.stepId.filter(item => item.success).map(i => i.data) // only successes
+```
+
+### From Paginated Step
+
+Paginated steps merge all pages server-side into a **single** result. The result is always a single envelope, NOT an array — even though multiple pages were fetched.
+
+```javascript
+sourceData.stepId = {
+  data: <merged response from all pages>,
+  success: true
+}
+
+// Access:
+sourceData.stepId.data              // the merged response
+sourceData.stepId.data.items        // if the API wraps results in .items
+
+// WRONG — do NOT treat paginated results as an array of envelopes:
+sourceData.stepId.map(...)          // FAILS — stepId is an object, not an array
+```
+
+## sourceData Structure
+
+`sourceData` is the cumulative state available to data selectors, variable expressions, and transforms:
+
+```javascript
+sourceData = {
+  // Original payload fields at ROOT level (NOT under .payload)
+  userId: "abc",
+  date: "2024-01-15",
+  companies: ["acme", "globex"],
+
+  // Previous step results, keyed by step ID
+  getUsers: { currentItem: {}, data: { users: [...] }, success: true },
+  fetchDetails: [
+    { currentItem: { id: 1 }, data: { name: "Alice" }, success: true },
+    { currentItem: { id: 2 }, data: { name: "Bob" }, success: true },
+  ],
+
+  // Current item (only within a loop step's config)
+  currentItem: { id: 1 },
+
+  // Runtime file map — see file-handling skill for full reference
+  __files__: { ... }
+}
+```
+
+Treat `sourceData.__files__` as read-only. See the file-handling skill for file detection, the `RuntimeExecutionFile` shape, aliasing rules, and how to access files in transforms.
+
+## Three Transformation Points
+
+| Point                            | When it runs             | Purpose                              |
+| -------------------------------- | ------------------------ | ------------------------------------ |
+| `dataSelector`                   | Before step executes     | Controls input + single vs loop mode |
+| Transform step (`transformCode`) | As the step itself       | Reshape data between request steps   |
+| `outputTransform`                | After all steps complete | Shape final tool output              |
+
+## Transform Steps
+
+Transform steps use `type: "transform"` with a `transformCode` function. They do NOT have: systemId, url, method, headers, body, queryParams, or pagination.
+
+Results are wrapped in the **same envelope** as request steps:
+
+```javascript
+sourceData.formatForInsert = { currentItem: <dataSelector output>, data: <transformCode result>, success: true }
+```
+
+If the step has a dataSelector returning an array, the transform runs once per item. See tool-building guidelines for when to use transform steps vs dataSelectors vs outputTransform.
+
+### Returning Files From a Transform
+
+Transform steps can produce files by returning a reserved `__files__` key in the result. See the file-handling skill for the full shape, aliasing rules, and access patterns.
+
+## Output Transform
+
+Final transformation shaping the tool's output. Runs after all steps complete. Can access cumulative step result data and payloads.
+
+### Requirements
+
+- Function signature: `(sourceData) => { ... }`
+- **Must have a return statement**
+- Async/await IS supported (same Deno sandbox as transform steps)
+- NEVER include newlines or tabs in the code string
+
+## JS Code Constraints (all transform points)
+
+Runs in a Deno subprocess sandbox:
+
+- 8192 MB memory limit, 10 min timeout (transforms/selectors), 3s timeout (pagination stops)
+- Async/await IS supported in transforms
+- No filesystem access — `--deny-read`, `--deny-write`
+- No subprocess spawning — `--deny-run`
+- Network access IS available (`--allow-net`) — transforms can make HTTP requests
+- Environment variables are sandboxed — only safe vars (PATH, HOME, DENO_DIR) are visible, no server secrets
+- JSON-serializable I/O only
+- Must return a value (undefined returns null)
+
+### Serialization Safety
+
+All JS code is stored as a JSON string, then MessagePack-serialized to the Deno subprocess, then `eval()`'d. Patterns that break across this pipeline:
+
+- **Regex literals with `/`** — `/https?:\/\/([^/]+)/` corrupts. Use `new URL(str).hostname` or `str.split("//")[1].split("/")[0]` instead.
+- **Backslash-heavy patterns** — `\n`, `\t`, `\\` in regex/strings get misinterpreted. Prefer string methods (`.split()`, `.indexOf()`, `.includes()`) over regex.
+- **Literal newlines** — code string must be single-line. Use `;` between statements.
+
+**Rule of thumb**: prefer string methods over regex. When regex is unavoidable, use `new RegExp("pattern")` constructor over `/pattern/` literals.

package/dist/skills/superglue/references/databases.md

@@ -13,7 +13,6 @@ Superglue supports both PostgreSQL and Microsoft SQL Server (including Azure SQL
   type: "request",
   systemId: "my_postgres_db",
   url: "postgres://<<my_postgres_db_user>>:<<my_postgres_db_password>>@<<my_postgres_db_host>>:<<my_postgres_db_port>>/<<my_postgres_db_database>>",
-  method: "POST",            // ignored for Postgres but required by schema
   body: '{"query": "SELECT * FROM users WHERE id = $1", "params": [<<userId>>]}'
 }
 ```
@@ -104,7 +103,6 @@ Use a data selector returning an array to execute a query per item:
   type: "request",
   systemId: "my_azure_sql",
   url: "mssql://<<my_azure_sql_user>>:<<my_azure_sql_password>>@<<my_azure_sql_host>>:<<my_azure_sql_port>>/<<my_azure_sql_database>>",
-  method: "POST",            // ignored for MSSQL but required by schema
   body: '{"query": "SELECT * FROM users WHERE id = @param1", "params": [<<userId>>]}'
 }
 ```
@@ -125,7 +123,13 @@ Both `mssql://` and `sqlserver://` protocols are supported. All credential varia
 For Azure SQL Database, use the fully qualified server name:
 
 ```
-mssql://myuser@myserver:mypassword@myserver.database.windows.net:1433/mydatabase
+mssql://myuser:mypassword@myserver.database.windows.net:1433/mydatabase
+```
+
+Note: Azure SQL usernames often include `@servername` suffix (e.g., `myuser@myserver`). When using in URLs, encode the @ as `%40`:
+
+```
+mssql://myuser%40myserver:mypassword@myserver.database.windows.net:1433/mydatabase
 ```
 
 You can include connection parameters as query strings:

package/dist/skills/superglue/references/file-handling.md

@@ -0,0 +1,160 @@
+# File Handling
+
+How the runtime detects, parses, and exposes files across all protocol strategies (HTTP, FTP, SFTP, SMB).
+
+## When Does a Response Become a File?
+
+Every response is read as raw bytes. The runtime decides whether it routes to `producedFiles` (i.e. `sourceData.__files__`) or inline `data` using a priority chain — **not** based on size alone:
+
+1. **Magic-byte detection** (primary) — the raw bytes are inspected for known binary signatures (ZIP `PK\x03\x04`, PDF `%PDF`, GZIP `\x1f\x8b`, plus internal ZIP structure checks for Excel/DOCX). If matched, the response is a file regardless of size or HTTP headers. A 1KB ZIP and a 500MB ZIP both go to `__files__`.
+2. **`Content-Disposition: attachment`** — any response with `attachment` or `filename=` in this header becomes a file, even if byte detection found structured text.
+3. **`application/octet-stream`** — explicit octet-stream MIME becomes a file, catching binary payloads whose bytes don't match a known signature.
+4. **25MB size fallback** (last resort) — only applies when byte detection returned `RAW` (unrecognized format), the response is larger than 25MB, **and** the `Content-Type` is not clearly text-like (`text/*`, `application/json`, `*+json`, `application/xml`, etc.). This is the only path where size matters.
+
+Structured text responses (JSON, XML, CSV, YAML, HTML) that pass none of the above checks go to `data` only — no file entry.
+
+| Classification      | Types                       | Where result lives                                             |
+| ------------------- | --------------------------- | -------------------------------------------------------------- |
+| **Binary**          | PDF, Excel, DOCX, ZIP, GZIP | `producedFiles` + `data` (extracted/parsed content)            |
+| **Structured text** | JSON, XML, CSV, YAML, HTML  | `data` only (parsed to JS objects)                             |
+| **RAW**             | Unrecognized                | File if attachment/octet-stream/25MB fallback, else raw string |
+
+Binary responses expose `stepFileKeys` and populate `sourceData.__files__`. Structured text responses only populate `sourceData.stepId.data`.
+
+File server `get` operations always produce files regardless of detected type — content goes into both `data` and `producedFiles`.
+
+## Auto-Parsing
+
+| Type  | Parsed result                                                 |
+| ----- | ------------------------------------------------------------- |
+| JSON  | Native JS object/array                                        |
+| CSV   | Array of objects (header row = keys)                          |
+| XML   | JS object tree                                                |
+| HTML  | JS object tree                                                |
+| YAML  | Native JS object                                              |
+| Excel | Object keyed by sheet name → array of row objects (see below) |
+| PDF   | `{ textContent, structuredContent: [{ page, text }] }`        |
+| DOCX  | Markdown string                                               |
+| ZIP   | Object keyed by entry filename, each value recursively parsed |
+| GZIP  | Recursively parsed inner content                              |
+| RAW   | UTF-8 string fallback                                         |
+
+Detection is heuristic for text formats. YAML requires `---`, `%YAML`, or multiple `key: value` lines to be recognized. Short or ambiguous YAML can fall through to RAW and appear as a plain string instead of a parsed object — both in `.data` and `.extracted`.
+
+### Excel rules
+
+- Excel is NEVER a flat array. It is always `{ "SheetName": [ {row}, ... ] }` — even for single-sheet files.
+- Access rows via `extracted.SheetName` or `extracted["Sheet1"]`, never `extracted[0]`.
+- When producing xlsx, use namespace `http://schemas.openxmlformats.org/spreadsheetml/2006/main` (not the truncated path).
+
+## file:: Reference Syntax
+
+Use `file::<alias>.<suffix>` in step config `body` / `content` fields:
+
+| Suffix       | Returns                     | Use case                                |
+| ------------ | --------------------------- | --------------------------------------- |
+| `.raw`       | Original bytes (Uint8Array) | SFTP/FTP put, HTTP raw body upload      |
+| `.base64`    | Base64-encoded string       | APIs expecting base64 inside JSON/XML   |
+| `.extracted` | Parsed content              | Text fields needing parsed file content |
+
+Rules:
+
+- Bare `file::<key>` without a suffix is **NOT valid** — causes a runtime error
+- File references are resolved **only in body/content fields**, not in headers, query params, or URLs
+- For multi-file steps, use bracket notation: `file::stepId["report.csv"].raw`
+- For loop iterations: `file::stepId[0].raw`
+- Important: base64 access in transforms is capped at 500MB by design, so base64 encoded binary in transform steps is only available for files smaller than this
+
+## File Aliasing
+
+When a step produces files, the runtime first chooses a **root alias**:
+
+- non-loop step: `stepId`
+- loop iteration: `stepId[N]`
+
+Then it decides whether that root needs a filename suffix:
+
+- if that step/iteration produced **exactly one file**, the alias is just the root
+- if that step/iteration produced **multiple files**, append `["filename.ext"]` to distinguish them
+
+Examples:
+
+- one direct download: `downloadReport`
+- one loop iteration download: `downloadReport[0]`
+- one batch step returning multiple files: `downloadReport["summary.csv"]`
+- one loop iteration returning multiple files: `downloadReport[0]["summary.csv"]`
+
+This is about **one file vs many files at that step/iteration**, not about whether the runtime knows the filename. File server `get` operations know the filename, but single-file outputs still use the plain step ID (or `stepId[N]`) alias.
+
+Loop steps (those with a `dataSelector` that iterates) always use numeric indices — never filenames — even when each iteration fetches a named file. Filename brackets only appear when a single step or iteration returns multiple files in one response.
+
+These aliases are used both in `file::` references and as keys in `sourceData.__files__`.
+
+## Accessing Files in Transforms
+
+Inside `transformCode` and `outputTransform`, use `sourceData.__files__` with the **runtime alias**:
+
+```javascript
+sourceData.__files__.downloadStep; // correct (step ID for single file)
+sourceData.__files__["doubled.csv"]; // wrong (local filename)
+```
+
+For multi-file aliases with bracket notation, prefer discovering aliases from `stepFileKeys` or
+`Object.keys(sourceData.__files__)` instead of hardcoding quoted alias strings inside the transform
+code string. This avoids JSON/JS escaping mistakes like:
+
+```javascript
+sourceData.__files__['fetchThreeFiles[".bashrc"]']; // brittle inside JSON-encoded transform strings
+```
+
+Prefer:
+
+```javascript
+(sourceData) => {
+  return (sourceData.fetchThreeFiles.stepFileKeys || []).map(function (alias) {
+    var f = sourceData.__files__[alias];
+    return {
+      alias: alias,
+      filename: f ? f.filename : null,
+      size: f ? f.size : null,
+    };
+  });
+};
+```
+
+`sourceData.__files__` is **read-only**. To produce new files from a transform, return them via a `__files__` key:
+
+```javascript
+(sourceData) => ({
+  summary: "processed",
+  __files__: {
+    "output.csv": {
+      filename: "output.csv",
+      contentType: "text/csv",
+      raw: new TextEncoder().encode("a,b\n1,2\n"),
+    },
+  },
+});
+```
+
+Everything except `__files__` becomes the step's `data`. Single-file transforms get the step ID as alias; multi-file transforms use bracket notation.
+
+### `raw` rules
+
+- `raw` accepts: `Uint8Array | ArrayBuffer | number[] | string | object | null`.
+  - **String** → UTF-8 encoded. Pass a JSON string for structured data: `raw: JSON.stringify(data)`.
+  - **Object / array of objects** → automatically JSON-stringified to UTF-8 bytes. Works, but prefer explicit `JSON.stringify` with a `text/csv` or `application/json` contentType for clarity.
+  - **`null` / `undefined`** → produces an empty file (0 bytes).
+  - **`Uint8Array` / `ArrayBuffer` / `number[]`** → used as-is for binary content.
+- For binary formats (xlsx, zip, pdf): pass through original bytes via `sourceData.__files__.stepId.raw` — never construct the internal file structure as a JS object.
+
+## stepFileKeys
+
+Each step result exposes `stepFileKeys` — an array of runtime aliases for files produced by that step:
+
+```javascript
+sourceData.downloadStep.stepFileKeys; // ["downloadStep"]
+sourceData.batchDownload[0].stepFileKeys; // ["batchDownload[0]"]
+```
+
+Use `stepFileKeys` to dynamically discover which file aliases are available from a step.

package/dist/skills/superglue/references/file-servers.md

@@ -12,10 +12,14 @@ smb://user:password@host/sharename/basePath  (port 445)
 smb://domain\user:password@host/sharename/   (domain auth)
 ```
 
-All support `<<variable>>` syntax in the URL.
+All support `<<variable>>` syntax in the URL. **For password-based auth, explicitly include credentials in the connection URL** using `<<systemId_username>>` and `<<systemId_password>>` placeholders (e.g. `sftp://<<mySystem_username>>:<<mySystem_password>>@host/path`). Authentication is never automatic. For SFTP private-key auth, use `credentials.privateKey`/`credentials.passphrase` instead of URL placeholders.
 
 SMB requires a share name as the first path segment. Additional segments become a base path prepended to all operation paths.
 
+## System Credentials
+
+Only store connection credentials: `host`, `port`, `username`, `password`, `privateKey`, `passphrase`, `domain` (SMB), `home_dir`. Paths and share names belong in the URL, not credentials.
+
 ## Path Handling
 
 **All operation paths are relative to the base path in the URL.** Do NOT use absolute filesystem paths.
@@ -63,19 +67,21 @@ JSON string — single operation or array of operations (batch):
 | `exists`  | `path`            | `{ exists: boolean, path }`                                             |
 | `stat`    | `path`            | File metadata or `{ exists: false }`                                    |
 
-### `get` — Auto-Parsing
+### `get` — Step File Keys
 
-Downloaded files are all parsed automatically:
+Downloaded files are added to the runtime file store. See the file-handling skill for the full reference on file detection, aliasing, and the `RuntimeExecutionFile` shape.
 
-- CSV, JSON, XML, Excel, PDF, etc. → parsed to JS objects
-- Falls back to UTF-8 string
-- SMB additionally detects binary files (returns `{ _binary: true, encoding: "base64", data: "..." }`)
+- `data` contains the auto-parsed content (CSV -> objects, JSON -> parsed, PDF -> structured, etc.)
+- the step result exposes `stepFileKeys`
+- later steps can reference downloaded files via `file::<stepId>.raw` (exact bytes), `file::<stepId>.base64` (base64 string), or `file::<stepId>.extracted` (parsed content)
+- for multi-file operations, use bracket notation: `file::<stepId>["report.csv"].raw`
 
 ### `put` — Content Handling
 
+- `RawFileBytes` (from `file::<key>.raw`) → written as exact original bytes
 - String → written as-is
 - Buffer → written directly
-- Object/Array → `JSON.stringify` with 2-space indent
+- Object/Array → JSON.stringified with 2-space indent
 
 ## Connection Details
 
@@ -104,33 +110,3 @@ For private systems via Secure Gateway, the host is `tunnelId.tunnel` (e.g., `my
 ## Return Value
 
 Single operation → result directly. Multiple operations → array of results.
-
-## Common Patterns
-
-### Download and process
-
-```json
-{ "operation": "get", "path": "incoming/<<(sourceData) => sourceData.currentItem.filename>>" }
-```
-
-### Upload results
-
-```json
-{
-  "operation": "put",
-  "path": "output/results.json",
-  "content": "<<(sourceData) => JSON.stringify(sourceData.processData.data)>>"
-}
-```
-
-### List then loop
-
-Step 1 lists files, step 2 uses data selector to loop over them:
-
-```javascript
-// Step 1: list files
-// body: {"operation": "list", "path": "incoming"}
-
-// Step 2 dataSelector: (sourceData) => sourceData.listFiles.data.filter(f => f.type === "file")
-// body: {"operation": "get", "path": "<<(sourceData) => sourceData.currentItem.path>>"}
-```