@superglue/cli 1.1.22 → 1.1.24

package/skills/superglue/references/file-handling.md

@@ -139,23 +139,49 @@ Prefer discovering aliases dynamically via `stepFileKeys`:
 
 ### Producing Files From Transforms
 
-Return a `__files__` key from a transform to produce new files:
+Return a `__files__` key from a transform to produce new files. The **key name is the filename** — content type is inferred from the extension. Values can be plain strings, bytes, or full file objects:
 
 ```javascript
+// Simplest: key = filename, value = string content (auto-encoded to bytes)
 (sourceData) => ({
-  summary: "processed",
   __files__: {
-    "output.csv": {
-      filename: "output.csv",
-      contentType: "text/csv",
-      raw: new TextEncoder().encode("a,b\n1,2\n"),
-    },
+    "report.csv": Papa.unparse(sourceData.step1.data),
   },
-});
+})
+
+// Multiple files with different types
+(sourceData) => ({
+  __files__: {
+    "data.csv": Papa.unparse(sourceData.step1.data),
+    "config.yaml": yaml.dump(sourceData.step1.data),
+  },
+  summary: { rowCount: sourceData.step1.data.length },
+})
 ```
 
+Supported value types for `__files__` entries:
+
+- **String** — auto-encoded to UTF-8 bytes. Filename and content type from the key.
+- **Uint8Array / ArrayBuffer** — used as-is. Filename and content type from the key.
+- **Object with `raw`** — `raw` is normalized (string/bytes/array), `filename` defaults to key if missing, `contentType` inferred from key extension if missing.
+- **Full file object** — `{ filename, contentType, raw }` for explicit control (backwards compatible).
+
 Everything except `__files__` becomes the step's `data`. Single-file transforms get the step ID as alias; multi-file transforms use bracket notation.
 
+**Available libraries in transforms:** The following libraries are available for file generation in any transform code (step transforms and output transforms):
+
+- `Papa` — [papaparse](https://www.papaparse.com/). Use `Papa.unparse(rows)` to generate CSV from an array of objects. Handles quoting, escaping, and special characters automatically.
+- `XLSX` — [SheetJS](https://sheetjs.com/). Use `XLSX.utils.json_to_sheet(data)` + `XLSX.write(wb, { type: "array", bookType: "xlsx" })` to generate Excel files.
+- `yaml` — [js-yaml](https://github.com/nodeca/js-yaml). Use `yaml.dump(obj)` to generate YAML.
+
+Example — CSV with Papa:
+
+```javascript
+(sourceData) => ({
+  __files__: { "report.csv": Papa.unparse(sourceData.step1.data) },
+});
+```
+
 **Auto-parsing of transform-produced files:** if `extracted` is omitted from the file object, the runtime runs the full detection and parsing pipeline on the raw bytes (same as for HTTP/file server responses). This means a CSV file produced with `raw: "a,b\n1,2"` will automatically get `extracted: [{a: "1", b: "2"}]`. To skip auto-parsing and keep the raw content as-is, set `extracted` explicitly (e.g. `extracted: null`).
 
 **`raw` field rules:**
@@ -166,6 +192,36 @@ Everything except `__files__` becomes the step's `data`. Single-file transforms
 - `null` / `undefined` — produces an empty file (0 bytes)
 - For binary formats (xlsx, zip, pdf): pass through original bytes via `sourceData.__files__.stepId.raw` — never construct the internal file structure as a JS object
 
+### Returning Files as Tool Output
+
+To make files downloadable from the tool's API response, set `outputFile: true` on the step that produces the file. The output transform is only for shaping JSON data — it has nothing to do with files.
+
+```json
+{
+  "steps": [
+    {
+      "id": "fetchData",
+      "config": { "type": "request", "url": "https://api.example.com/report" }
+    },
+    {
+      "id": "generateReport",
+      "config": {
+        "type": "transform",
+        "transformCode": "(sourceData) => ({ __files__: { \"report.csv\": Papa.unparse(sourceData.fetchData.data) } })"
+      },
+      "outputFile": true
+    }
+  ],
+  "outputTransform": "(sourceData) => ({ rowCount: sourceData.fetchData.data.length })"
+}
+```
+
+Only steps with `outputFile: true` have their files stored to S3 and returned as downloadable artifacts. Steps without the flag keep their files internal (for inter-step passing only).
+
+This works on any step type — transform steps, HTTP steps, SFTP/FTP steps. Just set `outputFile: true`.
+
+Use `sg run download <runId>` to download file artifacts from the CLI.
+
 ## Common Pitfalls
 
 - Using bare `file::<key>` without `.raw`, `.base64`, or `.extracted` suffix — causes a runtime error
@@ -174,6 +230,8 @@ Everything except `__files__` becomes the step's `data`. Single-file transforms
 - Hardcoding bracket aliases like `sourceData.__files__["step[\"file.csv\"]"]` — prefer `stepFileKeys` or `Object.keys(sourceData.__files__)` to discover aliases dynamically
 - Trying to construct binary file internals (xlsx, zip) as JS objects — always pass through original bytes via `.raw`
 - Accessing `sourceData.__files__` in step configs — `file::` syntax is for step configs, `sourceData.__files__` is for transform code
+- Manually base64-encoding file content with `btoa()` — never do this. Return raw bytes (Uint8Array) or strings directly. The runtime handles encoding automatically. Use `XLSX.write(wb, { type: "array" })` not `type: "buffer"` followed by manual btoa.
+- Putting file logic in the output transform — the output transform is for JSON data only. Use `outputFile: true` on the step instead.
 
 ## Error Recovery
 

package/skills/superglue/references/http.md

@@ -224,10 +224,15 @@ pagination: {
 
 **Paginated step results:** all pages merge into a single result. Access via `sourceData.stepId.data` — it is NOT an array of per-page results. Do not call `.map()` on a paginated step result. Returns single object if one result, array if multiple (unwrapped from single-element array).
 
+### File Output
+
+When an HTTP step downloads a file (PDF, CSV, binary response), set `outputFile: true` on the step to make it downloadable by the user. Without the flag, downloaded files stay internal (only accessible to subsequent steps via `file::` references).
+
 ## Common Pitfalls
 
 - Hardcoding base URLs that match the system's base URL instead of using `<<systemId_url>>`
 - Forgetting auth headers entirely — nothing is injected automatically, including for OAuth systems (only token refresh is automatic)
+- Forgetting `outputFile: true` on steps that download files the user wants
 - Putting pagination variables in the config but not setting a `pagination` block, or vice versa
 - Using `<<currentItem.id>>` — must use arrow function syntax: `<<(sourceData) => sourceData.currentItem.id>>`
 - Mixing multiple `<<>>` expressions in one body string when the API expects nested JSON — use a transform step instead

package/skills/superglue/references/odbc.md

@@ -0,0 +1,162 @@
+# ODBC
+
+## Step Configuration
+
+```typescript
+{
+  type: "request",
+  systemId: "my_sybase_db",
+  url: "odbc://<<my_sybase_db_username>>:<<my_sybase_db_password>>@<<my_sybase_db_host>>:<<my_sybase_db_port>>/<<my_sybase_db_database>>?DRIVER=<<my_sybase_db_driver>>",
+  body: '{"query": "SELECT * FROM users WHERE id = ?", "params": [<<userId>>]}'
+}
+```
+
+Configure the connection URL based on which credentials are stored in the system — use `sg system find` to check `storedCredentials` and reference them accordingly.
+
+`method`, `headers`, `queryParams`, and `pagination` are HTTP-only fields — omit them for database steps.
+
+### Fields
+
+| Field      | Required | Notes                                                          |
+| ---------- | -------- | -------------------------------------------------------------- |
+| `url`      | yes      | Connection string with `<<credential>>` placeholders           |
+| `body`     | yes      | JSON string with `query` and optional `params`                 |
+| `systemId` | no       | Links system credentials for `<<systemId_credKey>>` resolution |
+
+### URL Format
+
+```
+odbc://user:password@host:port/database?DRIVER=DriverName
+odbc://user:password@host:port/database?DRIVER=FreeTDS&TDS_Version=5.0
+```
+
+The path contains the **database name**. The ODBC driver is specified via the `DRIVER` query parameter (defaults to `FreeTDS` if omitted). This format is consistent with other superglue database strategies (postgres, mssql) where the path is always the database.
+
+**Supported drivers:**
+
+| Driver    | Database                     | Default Port |
+| --------- | ---------------------------- | ------------ |
+| `FreeTDS` | SAP ASE (Sybase), SQL Server | 5000         |
+
+Additional query parameters (beyond `DRIVER`) are passed through to the ODBC connection string as key=value pairs.
+
+### Body Format
+
+JSON string with `query` and optional `params` (or `values`):
+
+```json
+{
+  "query": "SELECT * FROM users WHERE age > ? AND status = ?",
+  "params": [25, "active"]
+}
+```
+
+## Authentication
+
+Credentials are embedded directly in the connection URL using `<<systemId_credentialKey>>` placeholders. Nothing is injected automatically.
+
+```
+odbc://<<my_db_username>>:<<my_db_password>>@<<my_db_host>>:<<my_db_port>>/<<my_db_database>>?DRIVER=<<my_db_driver>>
+```
+
+Credential variables are resolved at runtime before connecting.
+
+## ODBC Runtime Details
+
+### Schema Discovery
+
+superglue does not auto-introspect ODBC database schemas. Before building tools, always query the schema manually with `sg system call` to discover tables and columns.
+
+**SAP ASE (Sybase):**
+
+```json
+{
+  "query": "SELECT o.name AS table_name, c.name AS column_name, t.name AS data_type, CASE WHEN c.status & 8 = 8 THEN 'YES' ELSE 'NO' END AS is_nullable FROM sysobjects o JOIN syscolumns c ON o.id = c.id JOIN systypes t ON c.usertype = t.usertype WHERE o.type = 'U' ORDER BY o.name, c.colid"
+}
+```
+
+Always verify table and column names against the actual schema before building tools.
+
+### Parameterized Queries
+
+Always use parameterized queries with positional `?` placeholders — prevents SQL injection. Parameters are bound positionally from the `params` array.
+
+For numeric or boolean parameters in query params arrays, use unquoted placeholders: `[<<numericParam>>]` not `["<<numericParam>>"]`. The quotes force string resolution.
+
+```json
+{ "query": "SELECT * FROM orders WHERE name = ? AND customer_id = ?", "params": ["<<customerName>>", <<customerId>>] }
+```
+
+With expressions:
+
+```json
+{
+  "query": "INSERT INTO logs (message, level) VALUES (?, ?)",
+  "params": ["<<(sourceData) => sourceData.message>>", "error"]
+}
+```
+
+### Return Value
+
+Returns an array of row objects with column names as keys.
+
+- `SELECT`: array of matching rows
+- `INSERT/UPDATE/DELETE`: empty array (SAP ASE does not support `RETURNING` — query the data separately if needed)
+
+### Batch Operations
+
+Use a data selector returning an array to execute a query per item:
+
+```javascript
+// dataSelector: (sourceData) => sourceData.newUsers
+// body: {"query": "INSERT INTO users (name) VALUES (?)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
+```
+
+### Retry Behavior
+
+Default retries from server config. On final failure, error includes a truncated query preview.
+
+## SAP ASE (Sybase) Specifics
+
+### Differences from SQL Server
+
+SAP ASE uses Transact-SQL like SQL Server, but with key differences:
+
+- **No `OUTPUT` clause** — use `SELECT @@identity` after INSERT to get the last inserted identity value
+- **No `TOP` with `ORDER BY` guarantee** — use `SET ROWCOUNT n` before the query instead
+- **`CONVERT` syntax differs** — `CONVERT(target_type, expression)` not `CONVERT(expression, style)`
+- **String concatenation** — use `+` operator, not `CONCAT()` (unless ASE 16.0+)
+- **Date functions** — use `GETDATE()` not `CURRENT_TIMESTAMP`, `DATEADD`/`DATEDIFF` syntax matches SQL Server
+
+### TDS Protocol Version
+
+FreeTDS defaults to TDS 5.0 for Sybase connections. If you encounter protocol errors, you can specify the version via query parameter:
+
+```
+odbc://user:pass@host:5000/mydb?DRIVER=FreeTDS&TDS_Version=5.0
+```
+
+| TDS Version | Use Case                   |
+| ----------- | -------------------------- |
+| `5.0`       | SAP ASE / Sybase (default) |
+| `7.0`       | SQL Server 7.0+            |
+| `7.4`       | SQL Server 2012+           |
+
+## Common Pitfalls
+
+- Using `@param1`, `@param2` (MSSQL syntax) or `$1`, `$2` (PostgreSQL syntax) instead of positional `?` placeholders — ODBC uses `?` for all parameter binding
+- Forgetting that SAP ASE has no `OUTPUT`/`RETURNING` clause — mutations return empty arrays; query the data separately
+- Using `CONCAT()` on older ASE versions — use `+` for string concatenation instead
+- Driver name mismatch — the `DRIVER` query parameter must exactly match the section name in `odbcinst.ini` (case-sensitive). Use `DRIVER=FreeTDS`, not `DRIVER=freetds` or `DRIVER=FREETDS`
+- Accessing step result directly instead of via `.data` — database results follow the same envelope as all steps: `sourceData.stepId.data`
+
+## Error Recovery
+
+When an ODBC step fails and the cause is not obvious:
+
+1. **Read the error message** — superglue includes the a truncated query preview in the error. Check for syntax errors, missing columns, or type mismatches.
+2. **Verify connection credentials** — use `sg system find` to confirm `storedCredentials` keys match the placeholders in your URL. Test connectivity with a simple `SELECT 1` via `sg system call`.
+3. **Check database-side access** — connection refused or permission denied errors typically mean the database user lacks access. Verify with the user that the credentials have the required grants.
+4. **Check driver availability** — if the error mentions "driver not found", the `DRIVER` query parameter does not match any registered driver in `/etc/odbcinst.ini`. Verify the driver name is correct and the driver is installed in the Docker image.
+5. **Test the query directly** — use `sg system call` with a simplified version of the query to isolate whether the issue is the query itself, the parameters, or the connection.
+6. **Check TDS version** — protocol errors may indicate a TDS version mismatch. Try adding `?TDS_Version=5.0` to the query parameters for SAP ASE connections.

package/skills/superglue/references/sftp-smb.md

@@ -134,9 +134,14 @@ Single operation returns the result directly. Array of operations returns an arr
 
 Default retries from server config. On final failure, error includes the full operations list. Each retry creates a fresh connection.
 
+### File Output
+
+When the user wants to **download** a file fetched via SFTP/SMB/FTP, set `outputFile: true` on the step. This stores the file in S3 and returns a download link in the API response. Without the flag, fetched files stay internal (only accessible to subsequent steps via `file::` references).
+
 ## Common Pitfalls
 
 - Using absolute filesystem paths instead of paths relative to the URL base path
+- Forgetting `outputFile: true` on steps that download files the user wants — without it, the file is only available internally between steps
 - Forgetting that SMB requires a share name as the first URL path segment
 - Using `file::<key>` without a suffix in `put` content. Use `file::<key>.raw` to preserve original bytes
 - Missing the double-data access pattern for `get` results: `sourceData.stepId.data` gives you the parsed file content (through the step envelope)