@superglue/cli 1.1.21 → 1.1.22

package/skills/superglue/references/data-handling.md

@@ -1,303 +0,0 @@
-# Data Handling: Variables, Selectors & Transforms
-
-Most tool building and editing failures trace back to incorrect variable syntax, wrong data selector returns, or misunderstanding of the tool result envelope.
-
-## Variable Replacement (`<<>>` Syntax)
-
-The `<<expression>>` syntax injects dynamic values into step config fields (URL, headers, body, queryParams).
-
-### Simple Variables (top-level keys only)
-
-```
-<<userId>>           payload keys
-<<currentItem>>      whole current loop item
-<<page>>             pagination variable
-<<systemId_api_key>> system credential
-<<systemId_url>>     system base URL
-<<sg_auth_email>>    email of the authenticated user
-```
-
-**CRITICAL**: Simple `<<varName>>` only works for top-level keys. NO dots, NO nesting.
-
-- VALID: `<<userId>>`, `<<currentItem>>`, `<<page>>`, `<<stripe_api_key>>`, `<<stripe_url>>`, `<<sg_auth_email>>`
-- INVALID: `<<currentItem.id>>`, `<<sourceData.userId>>`, `<<user.name>>` — these FAIL at runtime
-
-### Arrow Function Expressions (for everything else)
-
-```
-<<(sourceData) => sourceData.currentItem.player_id>>
-<<(sourceData) => sourceData.getUsers.data.map(u => u.id)>>
-<<(sourceData) => JSON.stringify({ ids: sourceData.fetchUsers.data.map(u => u.id) })>>
-```
-
-### Resolution Order
-
-1. Direct lookup: check if expression is a top-level key in the merged variable object
-2. Arrow function: execute in Deno subprocess sandbox with full data as `sourceData`
-3. Failure: throws error listing available keys
-
-### Variable Sources
-
-All variables come from a single merged object:
-
-```javascript
-{
-  ...originalPayload,       // user-provided input
-  ...previousStepResults,   // keyed by stepId
-  ...systemCredentials,     // namespaced: systemId_credKey
-  ...systemUrls,            // namespaced: systemId_url
-  sg_auth_email,            // email of authenticated user (context variable)
-  currentItem,              // current loop item (if in a loop)
-  page, offset, cursor,     // pagination variables
-  limit, pageSize           // pagination size
-}
-```
-
-## Context Variables
-
-Context variables provide information about the execution environment:
-
-- `<<sg_auth_email>>` — Email address of the authenticated user who triggered the tool execution
-
-**Use case**: Personalize tool outputs based on who is running the tool. For example:
-
-- Different permissions in Tableau based on user
-- User-specific account lookups
-- Audit logging with user identity
-
-**Important**: `sg_auth_email` is NOT available in scheduled executions (cron jobs). If a tool references `<<sg_auth_email>>` and runs via scheduler, it will fail with a clear error message.
-
-### Return Type Handling
-
-- Objects/arrays returned from expressions → JSON.stringified automatically
-- Resolved values of `"undefined"` or `"null"` → stripped from headers and queryParams
-
-### Special: Basic Auth
-
-`"Authorization": "Basic <<user>>:<<pass>>"` is auto-converted to Base64. Do NOT manually encode.
-
-## Credential Resolution
-
-Credentials are available via `<<>>` syntax. Two sources, payload takes precedence:
-
-1. **System credentials**: Stored on System, namespaced as `<<systemId_credKey>>`
-
-   ```javascript
-   // System id="stripe", credentials={ api_key: "sk_..." }
-   // Available as: <<stripe_api_key>>
-   ```
-
-2. **Payload credentials**: Passed at runtime, accessed directly
-   ```javascript
-   // Payload: { user_access_token: "abc" }
-   // Available as: <<user_access_token>>
-   // Or: <<(sourceData) => sourceData.user_access_token>>
-   ```
-
-For OAuth systems, use `<<systemId_access_token>>` — the token value is auto-refreshed during tool execution, but the header itself must be present in the step config.
-
-## System URL Resolution
-
-System URLs are available via `<<systemId_url>>` syntax. This enables environment-agnostic tools:
-
-```javascript
-// System id="salesforce", url="https://mycompany.salesforce.com"
-// Available as: <<salesforce_url>>
-
-// Step config:
-{
-  "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account",
-  "headers": { "Authorization": "Bearer <<salesforce_access_token>>" }
-}
-```
-
-**Why use system URL variables?**
-
-- Same tool works across dev/prod environments without modification
-- When switching execution mode, the system URL resolves to the appropriate environment
-- Avoids hardcoding base URLs that differ between environments
-
-## Data Selectors
-
-Every step has a `dataSelector` — a JS function that controls execution mode and what `currentItem` is.
-
-### Returns OBJECT → Single Execution
-
-Step runs once. The object becomes `currentItem`.
-
-```javascript
-(sourceData) => ({ userId: sourceData.userId })
-(sourceData) => ({})   // run once with no specific input
-```
-
-Result stored as:
-
-```javascript
-sourceData.stepId = { currentItem: <object>, data: <response>, success: true }
-```
-
-### Returns ARRAY → Loop Execution
-
-Step runs once per array element. Each element becomes `currentItem`.
-
-```javascript
-(sourceData) => sourceData.getContacts.data.filter(c => c.active)
-(sourceData) => sourceData.userIds   // if userIds is an array from payload
-```
-
-Result stored as:
-
-```javascript
-sourceData.stepId = [
-  { currentItem: <item1>, data: <response1>, success: true },
-  { currentItem: <item2>, data: <response2>, success: true },
-  ...
-]
-```
-
-Empty arrays are valid — step just skips execution.
-
-### Accessing currentItem in Step Config
-
-```
-<<currentItem>>                                           // whole value
-<<(sourceData) => sourceData.currentItem.id>>             // property
-<<(sourceData) => sourceData.currentItem.name.toUpperCase()>>  // with transform
-```
-
-## Result Envelope
-
-**Every** step result (request or transform) is wrapped identically:
-
-### From Object Selector
-
-```javascript
-sourceData.stepId = {
-  currentItem: <selectorOutput>,
-  data: <apiResponse or transformResult>,
-  success: true
-}
-
-// Access:
-sourceData.stepId.data           // the response
-sourceData.stepId.currentItem    // what was passed in
-```
-
-### From Array Selector
-
-```javascript
-sourceData.stepId = [
-  { currentItem: <item1>, data: <response1>, success: true },
-  { currentItem: <item2>, data: <response2>, success: false, error: "..." },
-  ...
-]
-
-// Access:
-sourceData.stepId.map(item => item.data)                     // all responses
-sourceData.stepId.flatMap(item => item.data.results)          // nested arrays
-sourceData.stepId.filter(item => item.success).map(i => i.data) // only successes
-```
-
-### From Paginated Step
-
-Paginated steps merge all pages server-side into a **single** result. The result is always a single envelope, NOT an array — even though multiple pages were fetched.
-
-```javascript
-sourceData.stepId = {
-  data: <merged response from all pages>,
-  success: true
-}
-
-// Access:
-sourceData.stepId.data              // the merged response
-sourceData.stepId.data.items        // if the API wraps results in .items
-
-// WRONG — do NOT treat paginated results as an array of envelopes:
-sourceData.stepId.map(...)          // FAILS — stepId is an object, not an array
-```
-
-## sourceData Structure
-
-`sourceData` is the cumulative state available to data selectors, variable expressions, and transforms:
-
-```javascript
-sourceData = {
-  // Original payload fields at ROOT level (NOT under .payload)
-  userId: "abc",
-  date: "2024-01-15",
-  companies: ["acme", "globex"],
-
-  // Previous step results, keyed by step ID
-  getUsers: { currentItem: {}, data: { users: [...] }, success: true },
-  fetchDetails: [
-    { currentItem: { id: 1 }, data: { name: "Alice" }, success: true },
-    { currentItem: { id: 2 }, data: { name: "Bob" }, success: true },
-  ],
-
-  // Current item (only within a loop step's config)
-  currentItem: { id: 1 },
-
-  // Runtime file map — see file-handling skill for full reference
-  __files__: { ... }
-}
-```
-
-Treat `sourceData.__files__` as read-only. See the file-handling skill for file detection, the `RuntimeExecutionFile` shape, aliasing rules, and how to access files in transforms.
-
-## Three Transformation Points
-
-| Point                            | When it runs             | Purpose                              |
-| -------------------------------- | ------------------------ | ------------------------------------ |
-| `dataSelector`                   | Before step executes     | Controls input + single vs loop mode |
-| Transform step (`transformCode`) | As the step itself       | Reshape data between request steps   |
-| `outputTransform`                | After all steps complete | Shape final tool output              |
-
-## Transform Steps
-
-Transform steps use `type: "transform"` with a `transformCode` function. They do NOT have: systemId, url, method, headers, body, queryParams, or pagination.
-
-Results are wrapped in the **same envelope** as request steps:
-
-```javascript
-sourceData.formatForInsert = { currentItem: <dataSelector output>, data: <transformCode result>, success: true }
-```
-
-If the step has a dataSelector returning an array, the transform runs once per item. See tool-building guidelines for when to use transform steps vs dataSelectors vs outputTransform.
-
-### Returning Files From a Transform
-
-Transform steps can produce files by returning a reserved `__files__` key in the result. See the file-handling skill for the full shape, aliasing rules, and access patterns.
-
-## Output Transform
-
-Final transformation shaping the tool's output. Runs after all steps complete. Can access cumulative step result data and payloads.
-
-### Requirements
-
-- Function signature: `(sourceData) => { ... }`
-- **Must have a return statement**
-- Async/await IS supported (same Deno sandbox as transform steps)
-- NEVER include newlines or tabs in the code string
-
-## JS Code Constraints (all transform points)
-
-Runs in a Deno subprocess sandbox:
-
-- 8192 MB memory limit, 10 min timeout (transforms/selectors), 3s timeout (pagination stops)
-- Async/await IS supported in transforms
-- No filesystem access — `--deny-read`, `--deny-write`
-- No subprocess spawning — `--deny-run`
-- Network access IS available (`--allow-net`) — transforms can make HTTP requests
-- Environment variables are sandboxed — only safe vars (PATH, HOME, DENO_DIR) are visible, no server secrets
-- JSON-serializable I/O only
-- Must return a value (undefined returns null)
-
-### Serialization Safety
-
-All JS code is stored as a JSON string, then MessagePack-serialized to the Deno subprocess, then `eval()`'d. Patterns that break across this pipeline:
-
-- **Regex literals with `/`** — `/https?:\/\/([^/]+)/` corrupts. Use `new URL(str).hostname` or `str.split("//")[1].split("/")[0]` instead.
-- **Backslash-heavy patterns** — `\n`, `\t`, `\\` in regex/strings get misinterpreted. Prefer string methods (`.split()`, `.indexOf()`, `.includes()`) over regex.
-- **Literal newlines** — code string must be single-line. Use `;` between statements.
-
-**Rule of thumb**: prefer string methods over regex. When regex is unavoidable, use `new RegExp("pattern")` constructor over `/pattern/` literals.

package/skills/superglue/references/databases.md

@@ -1,253 +0,0 @@
-# Database Steps
-
-Superglue supports both PostgreSQL and Microsoft SQL Server (including Azure SQL).
-
----
-
-## PostgreSQL
-
-### Step Configuration
-
-```typescript
-{
-  type: "request",
-  systemId: "my_postgres_db",
-  url: "postgres://<<my_postgres_db_user>>:<<my_postgres_db_password>>@<<my_postgres_db_host>>:<<my_postgres_db_port>>/<<my_postgres_db_database>>",
-  body: '{"query": "SELECT * FROM users WHERE id = $1", "params": [<<userId>>]}'
-}
-```
-
-Configure the connection url depending on which credentials are saved in the stored db system - reference them accordingly.
-
-## URL Format
-
-```
-postgres://user:password@host:port/database
-postgresql://user:password@host:port/database
-```
-
-All credential variables are resolved before connecting. Trailing slashes and extra slashes before query strings are cleaned.
-
-## Body Format
-
-JSON string with `query` and optional `params` (or `values`):
-
-```json
-{ "query": "SELECT * FROM users WHERE age > $1 AND status = $2", "params": [25, "active"] }
-```
-
-### Parameterized Queries
-
-**Always use parameterized queries** with `$1`, `$2`, etc. — prevents SQL injection.
-
-```json
-{ "query": "SELECT * FROM orders WHERE customer_id = $1", "params": ["<<customerId>>"] }
-```
-
-With expressions:
-
-```json
-{
-  "query": "INSERT INTO logs (message, level) VALUES ($1, $2)",
-  "params": ["<<(sourceData) => sourceData.message>>", "error"]
-}
-```
-
-With RETURNING:
-
-```json
-{
-  "query": "INSERT INTO users (name, email) VALUES ($1, $2) RETURNING *",
-  "params": [
-    "<<(sourceData) => sourceData.currentItem.name>>",
-    "<<(sourceData) => sourceData.currentItem.email>>"
-  ]
-}
-```
-
-## Connection Pooling
-
-- Pools cached by connection string, max 10 clients, 5s connection timeout
-- Idle pools cleaned up automatically
-- SSL auto-configured: enabled (with `rejectUnauthorized: false`) unless connecting to localhost without explicit sslmode
-
-## Retry Logic
-
-Default retries from server config. On final failure, error includes query text and params.
-
-## Return Value
-
-Returns `result.rows` — array of row objects with column names as keys.
-
-- `SELECT`: array of matching rows
-- `INSERT/UPDATE/DELETE` with `RETURNING`: array of returned rows
-- Mutations without `RETURNING`: empty array
-
-### Batch Operations via Loop
-
-Use a data selector returning an array to execute a query per item:
-
-```javascript
-// dataSelector: (sourceData) => sourceData.newUsers
-// body: {"query": "INSERT INTO users (name) VALUES ($1)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
-```
-
----
-
-## Microsoft SQL Server / Azure SQL
-
-### Step Configuration
-
-```typescript
-{
-  type: "request",
-  systemId: "my_azure_sql",
-  url: "mssql://<<my_azure_sql_user>>:<<my_azure_sql_password>>@<<my_azure_sql_host>>:<<my_azure_sql_port>>/<<my_azure_sql_database>>",
-  body: '{"query": "SELECT * FROM users WHERE id = @param1", "params": [<<userId>>]}'
-}
-```
-
-Configure the connection url depending on which credentials are saved in the stored db system - reference them accordingly.
-
-### URL Format
-
-```
-mssql://user:password@host:port/database
-sqlserver://user:password@host:port/database
-```
-
-Both `mssql://` and `sqlserver://` protocols are supported. All credential variables are resolved before connecting.
-
-#### Azure SQL Database Connection
-
-For Azure SQL Database, use the fully qualified server name:
-
-```
-mssql://myuser:mypassword@myserver.database.windows.net:1433/mydatabase
-```
-
-Note: Azure SQL usernames often include `@servername` suffix (e.g., `myuser@myserver`). When using in URLs, encode the @ as `%40`:
-
-```
-mssql://myuser%40myserver:mypassword@myserver.database.windows.net:1433/mydatabase
-```
-
-You can include connection parameters as query strings:
-
-```
-mssql://user:password@host:port/database?encrypt=true&trustServerCertificate=false
-```
-
-**Connection Parameters:**
-
-- `encrypt` (default: `true`) - Enable/disable encryption (required for Azure SQL)
-- `trustServerCertificate` (default: `false`) - Whether to trust the server certificate
-- `database` - Database name (can be in URL path or query parameter)
-
-### Body Format
-
-JSON string with `query` and optional `params` (or `values`):
-
-```json
-{
-  "query": "SELECT * FROM users WHERE age > @param1 AND status = @param2",
-  "params": [25, "active"]
-}
-```
-
-### Parameterized Queries
-
-**Always use parameterized queries** with `@param1`, `@param2`, etc. — prevents SQL injection.
-
-```json
-{ "query": "SELECT * FROM orders WHERE customer_id = @param1", "params": ["<<customerId>>"] }
-```
-
-With expressions:
-
-```json
-{
-  "query": "INSERT INTO logs (message, level) VALUES (@param1, @param2)",
-  "params": ["<<(sourceData) => sourceData.message>>", "error"]
-}
-```
-
-With OUTPUT (MSSQL equivalent of RETURNING):
-
-```json
-{
-  "query": "INSERT INTO users (name, email) OUTPUT INSERTED.* VALUES (@param1, @param2)",
-  "params": [
-    "<<(sourceData) => sourceData.currentItem.name>>",
-    "<<(sourceData) => sourceData.currentItem.email>>"
-  ]
-}
-```
-
-**Parameter Naming**: MSSQL uses named parameters with `@` prefix. The params array maps positionally: `params[0]` → `@param1`, `params[1]` → `@param2`, etc.
-
-### Connection Pooling
-
-- Pools cached by connection string, max 10 clients per pool
-- 5s connection timeout, configurable request timeout
-- Idle pools cleaned up automatically
-- SSL/TLS encryption enabled by default for Azure SQL
-
-### Retry Logic
-
-Default retries from server config. On final failure, error includes query text and params.
-
-### Return Value
-
-Returns the recordset — array of row objects with column names as keys.
-
-- `SELECT`: array of matching rows
-- `INSERT/UPDATE/DELETE` with `OUTPUT`: array of returned rows
-- Mutations without `OUTPUT`: empty array
-
-### Batch Operations via Loop
-
-Use a data selector returning an array to execute a query per item:
-
-```javascript
-// dataSelector: (sourceData) => sourceData.newUsers
-// body: {"query": "INSERT INTO users (name) VALUES (@param1)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
-```
-
-### Common Patterns
-
-**Simple SELECT:**
-
-```json
-{
-  "query": "SELECT id, name, email FROM users WHERE status = @param1",
-  "params": ["active"]
-}
-```
-
-**INSERT with OUTPUT:**
-
-```json
-{
-  "query": "INSERT INTO users (name, email, created_at) OUTPUT INSERTED.id, INSERTED.created_at VALUES (@param1, @param2, GETUTCDATE())",
-  "params": ["John Doe", "john@example.com"]
-}
-```
-
-**UPDATE with OUTPUT:**
-
-```json
-{
-  "query": "UPDATE users SET last_login = GETUTCDATE() OUTPUT INSERTED.id, INSERTED.last_login WHERE id = @param1",
-  "params": [123]
-}
-```
-
-**Stored Procedure Call:**
-
-```json
-{
-  "query": "EXEC dbo.GetUserById @param1",
-  "params": [123]
-}
-```

package/skills/superglue/references/file-servers.md

@@ -1,112 +0,0 @@
-# File Server Steps (FTP / SFTP / SMB)
-
-FTP, SFTP, and SMB share an identical operation interface. Only the URL protocol and connection details differ.
-
-## URL Formats
-
-```
-ftp://user:password@host:port/basePath       (port 21)
-ftps://user:password@host:port/basePath      (port 21, TLS)
-sftp://user:password@host:port/basePath      (port 22)
-smb://user:password@host/sharename/basePath  (port 445)
-smb://domain\user:password@host/sharename/   (domain auth)
-```
-
-All support `<<variable>>` syntax in the URL. **For password-based auth, explicitly include credentials in the connection URL** using `<<systemId_username>>` and `<<systemId_password>>` placeholders (e.g. `sftp://<<mySystem_username>>:<<mySystem_password>>@host/path`). Authentication is never automatic. For SFTP private-key auth, use `credentials.privateKey`/`credentials.passphrase` instead of URL placeholders.
-
-SMB requires a share name as the first path segment. Additional segments become a base path prepended to all operation paths.
-
-## System Credentials
-
-Only store connection credentials: `host`, `port`, `username`, `password`, `privateKey`, `passphrase`, `domain` (SMB), `home_dir`. Paths and share names belong in the URL, not credentials.
-
-## Path Handling
-
-**All operation paths are relative to the base path in the URL.** Do NOT use absolute filesystem paths.
-
-If the system URL is `sftp://user:pass@host/home/sftptest/uploads`:
-
-- `report.csv` → `/home/sftptest/uploads/report.csv`
-- `subdir/data.csv` → `/home/sftptest/uploads/subdir/data.csv`
-
-Do NOT use `/home/sftptest/uploads/report.csv` — the base path is already set in the URL.
-
-If the URL has no base path (e.g., `sftp://user:pass@host`), operation paths are used as-is from the server root.
-
-## Body Format
-
-JSON string — single operation or array of operations (batch):
-
-### Single
-
-```json
-{ "operation": "get", "path": "data/report.csv" }
-```
-
-### Batch (sequential, same connection)
-
-```json
-[
-  { "operation": "mkdir", "path": "backup" },
-  { "operation": "get", "path": "data/report.csv" },
-  { "operation": "put", "path": "backup/report.csv", "content": "data here" }
-]
-```
-
-## Supported Operations
-
-| Operation | Required fields   | Returns                                                                 |
-| --------- | ----------------- | ----------------------------------------------------------------------- |
-| `list`    | `path`            | Array of `{ name, path, size, type, modifyTime, ... }`                  |
-| `get`     | `path`            | Auto-parsed file content (CSV→objects, JSON→parsed, etc.) or raw string |
-| `put`     | `path`, `content` | `{ success, message, size }`                                            |
-| `delete`  | `path`            | `{ success, message }`                                                  |
-| `rename`  | `path`, `newPath` | `{ success, message }`                                                  |
-| `mkdir`   | `path`            | `{ success, message }`                                                  |
-| `rmdir`   | `path`            | `{ success, message }`                                                  |
-| `exists`  | `path`            | `{ exists: boolean, path }`                                             |
-| `stat`    | `path`            | File metadata or `{ exists: false }`                                    |
-
-### `get` — Step File Keys
-
-Downloaded files are added to the runtime file store. See the file-handling skill for the full reference on file detection, aliasing, and the `RuntimeExecutionFile` shape.
-
-- `data` contains the auto-parsed content (CSV -> objects, JSON -> parsed, PDF -> structured, etc.)
-- the step result exposes `stepFileKeys`
-- later steps can reference downloaded files via `file::<stepId>.raw` (exact bytes), `file::<stepId>.base64` (base64 string), or `file::<stepId>.extracted` (parsed content)
-- for multi-file operations, use bracket notation: `file::<stepId>["report.csv"].raw`
-
-### `put` — Content Handling
-
-- `RawFileBytes` (from `file::<key>.raw`) → written as exact original bytes
-- String → written as-is
-- Buffer → written directly
-- Object/Array → JSON.stringified with 2-space indent
-
-## Connection Details
-
-### SFTP
-
-Uses `ssh2-sftp-client`. Supports password + private key auth (`credentials.privateKey`, `credentials.passphrase`).
-
-### FTP/FTPS
-
-Uses `basic-ftp`. FTPS uses TLS with `rejectUnauthorized: false`.
-
-### SMB
-
-Uses `@awo00/smb2`. Connection lifecycle: create client → authenticate (domain optional) → connect tree (share) → execute → disconnect/logoff/close. Entire operation wrapped in timeout race.
-
-All paths use forward slashes (`/`) — SMB library handles Windows conversion internally.
-
-Credentials must be injected into the URL using placeholders:
-
-```
-smb://<<systemId_username>>:<<systemId_password>>@fileserver.example.com/ShareName
-```
-
-For private systems via Secure Gateway, the host is `tunnelId.tunnel` (e.g., `my_tunnel.tunnel`).
-
-## Return Value
-
-Single operation → result directly. Multiple operations → array of results.