@superglue/cli 1.1.20 → 1.1.22

package/skills/superglue/references/file-handling.md

@@ -1,128 +1,145 @@
 # File Handling
 
-How the runtime detects, parses, and exposes files across all protocol strategies (HTTP, FTP, SFTP, SMB).
+How the runtime detects, parses, and exposes files across all protocol strategies (HTTP, SFTP/FTP, SMB).
 
-## When Does a Response Become a File?
+## Step Configuration
 
-Every response is read as raw bytes. The runtime decides whether it routes to `producedFiles` (i.e. `sourceData.__files__`) or inline `data` using a priority chain — **not** based on size alone:
+File handling is not a standalone step type — it is a cross-cutting runtime behavior that applies to any step that produces or consumes files. The `file::<alias>.<suffix>` syntax is used in step config `body` / `content` fields to reference files from previous steps or user uploads.
 
-1. **Magic-byte detection** (primary) — the raw bytes are inspected for known binary signatures (ZIP `PK\x03\x04`, PDF `%PDF`, GZIP `\x1f\x8b`, plus internal ZIP structure checks for Excel/DOCX). If matched, the response is a file regardless of size or HTTP headers. A 1KB ZIP and a 500MB ZIP both go to `__files__`.
-2. **`Content-Disposition: attachment`** — any response with `attachment` or `filename=` in this header becomes a file, even if byte detection found structured text.
-3. **`application/octet-stream`** — explicit octet-stream MIME becomes a file, catching binary payloads whose bytes don't match a known signature.
-4. **25MB size fallback** (last resort) — only applies when byte detection returned `RAW` (unrecognized format), the response is larger than 25MB, **and** the `Content-Type` is not clearly text-like (`text/*`, `application/json`, `*+json`, `application/xml`, etc.). This is the only path where size matters.
+```typescript
+// Reference a file downloaded by a previous step
+{ "body": "file::downloadStep.raw" }
 
-Structured text responses (JSON, XML, CSV, YAML, HTML) that pass none of the above checks go to `data` only — no file entry.
+// Reference a user-uploaded file
+{ "body": '{"fileBase64": "file::document.base64"}' }
 
-| Classification      | Types                       | Where result lives                                             |
-| ------------------- | --------------------------- | -------------------------------------------------------------- |
-| **Binary**          | PDF, Excel, DOCX, ZIP, GZIP | `producedFiles` + `data` (extracted/parsed content)            |
-| **Structured text** | JSON, XML, CSV, YAML, HTML  | `data` only (parsed to JS objects)                             |
-| **RAW**             | Unrecognized                | File if attachment/octet-stream/25MB fallback, else raw string |
+// Reference a specific file from a multi-file step
+{ "content": 'file::batchDownload["report.csv"].raw' }
+```
 
-Binary responses expose `stepFileKeys` and populate `sourceData.__files__`. Structured text responses only populate `sourceData.stepId.data`.
+### file:: Reference Syntax
 
-File server `get` operations always produce files regardless of detected type — content goes into both `data` and `producedFiles`.
+Use `file::<alias>.<suffix>` in step config `body` / `content` fields:
 
-## Auto-Parsing
+| Suffix       | Returns                     | Use case                                |
+| ------------ | --------------------------- | --------------------------------------- |
+| `.raw`       | Original bytes (Uint8Array) | SFTP/FTP put, HTTP raw body upload      |
+| `.base64`    | Base64-encoded string       | APIs expecting base64 inside JSON/XML   |
+| `.extracted` | Parsed content              | Text fields needing parsed file content |
 
-| Type  | Parsed result                                                 |
-| ----- | ------------------------------------------------------------- |
-| JSON  | Native JS object/array                                        |
-| CSV   | Array of objects (header row = keys)                          |
-| XML   | JS object tree                                                |
-| HTML  | JS object tree                                                |
-| YAML  | Native JS object                                              |
-| Excel | Object keyed by sheet name → array of row objects (see below) |
-| PDF   | `{ textContent, structuredContent: [{ page, text }] }`        |
-| DOCX  | Markdown string                                               |
-| ZIP   | Object keyed by entry filename, each value recursively parsed |
-| GZIP  | Recursively parsed inner content                              |
-| RAW   | UTF-8 string fallback                                         |
+Constraints:
 
-Detection is heuristic for text formats. YAML requires `---`, `%YAML`, or multiple `key: value` lines to be recognized. Short or ambiguous YAML can fall through to RAW and appear as a plain string instead of a parsed object — both in `.data` and `.extracted`.
+- Bare `file::<key>` without a suffix is invalid and causes a runtime error
+- File references resolve only in body/content fields, not in headers, query params, or URLs
+- Multi-file steps use bracket notation: `file::stepId["report.csv"].raw`
+- Loop iterations use numeric index: `file::stepId[0].raw`
 
-### Excel rules
+## Authentication
 
-- Excel is NEVER a flat array. It is always `{ "SheetName": [ {row}, ... ] }` — even for single-sheet files.
-- Access rows via `extracted.SheetName` or `extracted["Sheet1"]`, never `extracted[0]`.
-- When producing xlsx, use namespace `http://schemas.openxmlformats.org/spreadsheetml/2006/main` (not the truncated path).
+N/A — file handling is a runtime behavior, not a protocol. Authentication is handled by the protocol skill that produces or consumes the file (http, sftp-smb, etc.).
 
-## file:: Reference Syntax
+## File Handling Runtime Details
 
-Use `file::<alias>.<suffix>` in step config `body` / `content` fields:
+### Detection Priority
 
-| Suffix       | Returns                     | Use case                                |
-| ------------ | --------------------------- | --------------------------------------- |
-| `.raw`       | Original bytes (Uint8Array) | SFTP/FTP put, HTTP raw body upload      |
-| `.base64`    | Base64-encoded string       | APIs expecting base64 inside JSON/XML   |
-| `.extracted` | Parsed content              | Text fields needing parsed file content |
+Every response is read as raw bytes. The runtime classifies it using this priority chain:
 
-Rules:
+1. **Magic-byte detection** — known binary signatures (ZIP `PK\x03\x04`, PDF `%PDF`, GZIP `\x1f\x8b`, plus internal ZIP structure checks for Excel/DOCX)
+2. **`Content-Disposition: attachment`** — `attachment` or `filename=` in this header forces file treatment
+3. **`application/octet-stream`** — explicit octet-stream MIME catches unlabeled binary payloads
+4. **25MB size fallback** — only when byte detection returned `RAW`, the response exceeds 25MB, and `Content-Type` is not text-like (`text/*`, `application/json`, `*+json`, `application/xml`)
 
-- Bare `file::<key>` without a suffix is **NOT valid** — causes a runtime error
-- File references are resolved **only in body/content fields**, not in headers, query params, or URLs
-- For multi-file steps, use bracket notation: `file::stepId["report.csv"].raw`
-- For loop iterations: `file::stepId[0].raw`
-- Important: base64 access in transforms is capped at 500MB by design, so base64 encoded binary in transform steps is only available for files smaller than this
+File server `get` operations always produce files regardless of detected type.
 
-## File Aliasing
+### Classification
 
-When a step produces files, the runtime first chooses a **root alias**:
+| Classification      | Types                       | Destination                                                    |
+| ------------------- | --------------------------- | -------------------------------------------------------------- |
+| **Binary**          | PDF, Excel, DOCX, ZIP, GZIP | `producedFiles` + `data` (extracted/parsed content)            |
+| **Structured text** | JSON, XML, CSV, YAML, HTML  | `data` only (parsed to JS objects)                             |
+| **RAW**             | Unrecognized                | File if attachment/octet-stream/25MB fallback, else raw string |
 
-- non-loop step: `stepId`
-- loop iteration: `stepId[N]`
+Binary responses expose `stepFileKeys` and populate `sourceData.__files__`. Structured text responses only populate `sourceData.stepId.data`.
+
+### Auto-Parsing
+
+| Type  | Parsed result                                                  |
+| ----- | -------------------------------------------------------------- |
+| JSON  | Native JS object/array                                         |
+| CSV   | Array of objects (header row = keys)                           |
+| XML   | JS object tree                                                 |
+| HTML  | JS object tree                                                 |
+| YAML  | Native JS object                                               |
+| Excel | Object keyed by sheet name, each value is array of row objects |
+| PDF   | `{ textContent, structuredContent: [{ page, text }] }`         |
+| DOCX  | Markdown string                                                |
+| ZIP   | Object keyed by entry filename, each value recursively parsed  |
+| GZIP  | Recursively parsed inner content                               |
+| RAW   | UTF-8 string fallback                                          |
 
-Then it decides whether that root needs a filename suffix:
+**Excel rules:**
 
-- if that step/iteration produced **exactly one file**, the alias is just the root
-- if that step/iteration produced **multiple files**, append `["filename.ext"]` to distinguish them
+- Excel is never a flat array. It is always `{ "SheetName": [ {row}, ... ] }` — even for single-sheet files
+- Access rows via `extracted.SheetName` or `extracted["Sheet1"]`, never `extracted[0]`
+- When producing xlsx, use namespace `http://schemas.openxmlformats.org/spreadsheetml/2006/main` (not the truncated path)
 
-Examples:
+### File Aliasing
 
-- one direct download: `downloadReport`
-- one loop iteration download: `downloadReport[0]`
-- one batch step returning multiple files: `downloadReport["summary.csv"]`
-- one loop iteration returning multiple files: `downloadReport[0]["summary.csv"]`
+When a step produces files, the runtime assigns an alias:
 
-This is about **one file vs many files at that step/iteration**, not about whether the runtime knows the filename. File server `get` operations know the filename, but single-file outputs still use the plain step ID (or `stepId[N]`) alias.
+| Scenario                    | Alias format                |
+| --------------------------- | --------------------------- |
+| Single file, non-loop       | `stepId`                    |
+| Single file, loop iteration | `stepId[N]`                 |
+| Multiple files, non-loop    | `stepId["filename.ext"]`    |
+| Multiple files, loop        | `stepId[N]["filename.ext"]` |
 
-Loop steps (those with a `dataSelector` that iterates) always use numeric indices — never filenames — even when each iteration fetches a named file. Filename brackets only appear when a single step or iteration returns multiple files in one response.
+Loop steps always use numeric indices — never filenames — even when each iteration fetches a named file. Filename brackets only appear when a single step or iteration returns multiple files in one response.
 
 These aliases are used both in `file::` references and as keys in `sourceData.__files__`.
 
-## Accessing Files in Transforms
+### stepFileKeys
 
-Inside `transformCode` and `outputTransform`, use `sourceData.__files__` with the **runtime alias**:
+Each step result exposes `stepFileKeys` — an array of runtime aliases for files produced by that step:
 
 ```javascript
-sourceData.__files__.downloadStep; // correct (step ID for single file)
-sourceData.__files__["doubled.csv"]; // wrong (local filename)
+sourceData.downloadStep.stepFileKeys; // ["downloadStep"]
+sourceData.batchDownload[0].stepFileKeys; // ["batchDownload[0]"]
 ```
 
-For multi-file aliases with bracket notation, prefer discovering aliases from `stepFileKeys` or
-`Object.keys(sourceData.__files__)` instead of hardcoding quoted alias strings inside the transform
-code string. This avoids JSON/JS escaping mistakes like:
+### Accessing Files in Transforms
+
+Inside `transformCode` and `outputTransform`, use `sourceData.__files__` with the runtime alias. Treat `sourceData.__files__` as read-only — the runtime rebuilds it after every step.
 
 ```javascript
-sourceData.__files__['fetchThreeFiles[".bashrc"]']; // brittle inside JSON-encoded transform strings
+sourceData.__files__.downloadStep; // single-file step
+sourceData.__files__["stepId[0]"]; // loop iteration
 ```
 
-Prefer:
+Available properties on each file object: `filename`, `contentType`, `size`, `raw`, `extracted`, `fileType`, `parseError`.
+
+**base64 access:** `base64` is a lazy getter — it is not stored on the file object by default. Access it directly:
+
+```javascript
+sourceData.__files__.myStep.base64; // computes and returns base64 string on first access
+```
+
+The base64 value is computed on first access and cached for subsequent reads. It will not appear in `Object.keys()` or `JSON.stringify()` output (non-enumerable). Throws if the file exceeds 500MB — use `.raw` for large files.
+
+Prefer discovering aliases dynamically via `stepFileKeys`:
 
 ```javascript
 (sourceData) => {
   return (sourceData.fetchThreeFiles.stepFileKeys || []).map(function (alias) {
     var f = sourceData.__files__[alias];
-    return {
-      alias: alias,
-      filename: f ? f.filename : null,
-      size: f ? f.size : null,
-    };
+    return { alias: alias, filename: f ? f.filename : null, size: f ? f.size : null };
   });
 };
 ```
 
-`sourceData.__files__` is **read-only**. To produce new files from a transform, return them via a `__files__` key:
+### Producing Files From Transforms
+
+Return a `__files__` key from a transform to produce new files:
 
 ```javascript
 (sourceData) => ({
@@ -139,22 +156,31 @@ Prefer:
 
 Everything except `__files__` becomes the step's `data`. Single-file transforms get the step ID as alias; multi-file transforms use bracket notation.
 
-### `raw` rules
+**Auto-parsing of transform-produced files:** if `extracted` is omitted from the file object, the runtime runs the full detection and parsing pipeline on the raw bytes (same as for HTTP/file server responses). This means a CSV file produced with `raw: "a,b\n1,2"` will automatically get `extracted: [{a: "1", b: "2"}]`. To skip auto-parsing and keep the raw content as-is, set `extracted` explicitly (e.g. `extracted: null`).
 
-- `raw` accepts: `Uint8Array | ArrayBuffer | number[] | string | object | null`.
-  - **String** → UTF-8 encoded. Pass a JSON string for structured data: `raw: JSON.stringify(data)`.
-  - **Object / array of objects** → automatically JSON-stringified to UTF-8 bytes. Works, but prefer explicit `JSON.stringify` with a `text/csv` or `application/json` contentType for clarity.
-  - **`null` / `undefined`** → produces an empty file (0 bytes).
-  - **`Uint8Array` / `ArrayBuffer` / `number[]`** → used as-is for binary content.
-- For binary formats (xlsx, zip, pdf): pass through original bytes via `sourceData.__files__.stepId.raw` — never construct the internal file structure as a JS object.
+**`raw` field rules:**
 
-## stepFileKeys
+- `Uint8Array` / `ArrayBuffer` / typed arrays — used as-is for binary content
+- String — UTF-8 encoded. Pass a JSON string for structured data: `raw: JSON.stringify(data)`
+- Object / array — auto JSON-stringified to UTF-8 bytes. Prefer explicit `JSON.stringify` with an appropriate `contentType` for clarity
+- `null` / `undefined` — produces an empty file (0 bytes)
+- For binary formats (xlsx, zip, pdf): pass through original bytes via `sourceData.__files__.stepId.raw` — never construct the internal file structure as a JS object
 
-Each step result exposes `stepFileKeys` — an array of runtime aliases for files produced by that step:
+## Common Pitfalls
 
-```javascript
-sourceData.downloadStep.stepFileKeys; // ["downloadStep"]
-sourceData.batchDownload[0].stepFileKeys; // ["batchDownload[0]"]
-```
+- Using bare `file::<key>` without `.raw`, `.base64`, or `.extracted` suffix — causes a runtime error
+- Placing file references in headers or query params instead of body/content fields — they only resolve in body
+- Expecting Excel parsed output to be a flat array — it is always `{ "SheetName": [...] }` even for single-sheet files
+- Hardcoding bracket aliases like `sourceData.__files__["step[\"file.csv\"]"]` — prefer `stepFileKeys` or `Object.keys(sourceData.__files__)` to discover aliases dynamically
+- Trying to construct binary file internals (xlsx, zip) as JS objects — always pass through original bytes via `.raw`
+- Accessing `sourceData.__files__` in step configs — `file::` syntax is for step configs, `sourceData.__files__` is for transform code
+
+## Error Recovery
+
+When a file-related step fails:
 
-Use `stepFileKeys` to dynamically discover which file aliases are available from a step.
+1. **File input failure** - Ask the user whether the file is present in the session state and verify whether you can access the file at all
+2. **Check the file:: reference** — verify the alias matches a step that actually produces files. Use `stepFileKeys` on the producing step's result to see available aliases.
+3. **Verify the suffix** — bare `file::<key>` without `.raw`, `.base64`, or `.extracted` always fails. Choose the suffix based on how the consuming step needs the data.
+4. **Check the producing step** — if `stepFileKeys` is empty or undefined, the producing step may not have returned a file. Verify the response is binary or has `Content-Disposition: attachment` (for HTTP), or that the operation is `get` (for file servers).
+5. **Check aliasing for loops** — loop iterations use numeric indices (`stepId[0]`), not filenames. If the alias doesn't match, run the producing step and inspect the result to see the actual aliases assigned.

package/skills/superglue/references/graphql.md

@@ -0,0 +1,131 @@
+# GraphQL
+
+GraphQL APIs use HTTP POST under the hood. This skill covers the GraphQL-specific conventions; the http skill covers the underlying HTTP runtime behavior.
+
+## Step Configuration
+
+```typescript
+{
+  type: "request",
+  systemId: "my_graphql_api",
+  url: "<<my_graphql_api_url>>/graphql",
+  method: "POST",
+  headers: {
+    "Authorization": "Bearer <<my_graphql_api_access_token>>",
+    "Content-Type": "application/json"
+  },
+  body: '{"query": "query GetUser($id: ID!) { user(id: $id) { name email } }", "variables": {"id": "<<userId>>"}}'
+}
+```
+
+### Fields
+
+| Field      | Required | Notes                                                  |
+| ---------- | -------- | ------------------------------------------------------ |
+| `url`      | yes      | GraphQL endpoint — typically `/graphql`                |
+| `method`   | yes      | Always `POST`                                          |
+| `headers`  | yes      | Must include `Content-Type: application/json` and auth |
+| `body`     | yes      | JSON string with `query` and optional `variables`      |
+| `systemId` | no       | Links system credentials and URL                       |
+
+### Body Format
+
+The body is always a JSON string with:
+
+```json
+{
+  "query": "query { ... }",
+  "variables": { ... }
+}
+```
+
+For dynamic variables, use `<<>>` expressions:
+
+```json
+{
+  "query": "query GetOrders($customerId: ID!, $limit: Int) { orders(customerId: $customerId, first: $limit) { id total } }",
+  "variables": {
+    "customerId": "<<customerId>>",
+    "limit": "<<(sourceData) => sourceData.pageSize || 50>>"
+  }
+}
+```
+
+For complex variable objects, use a preceding transform step to build the variables, then reference with `<<(sourceData) => JSON.stringify(sourceData.prepStep.data)>>` as the entire body.
+
+## Authentication
+
+Same as HTTP — credentials are placed in headers using `<<systemId_credentialKey>>` syntax. Nothing is injected automatically.
+
+## GraphQL Runtime Details
+
+### Queries vs Mutations
+
+- **Queries** (read-only): set `modify: false` on the step (this is the default)
+- **Mutations** (write): set `modify: true`
+
+Both use `method: "POST"` — do not rely on HTTP method to distinguish read vs write.
+
+### Response Shape
+
+GraphQL APIs return `{ "data": { ... }, "errors": [ ... ] }`. The superglue step result wraps this in the standard envelope:
+
+```javascript
+sourceData.stepId.data; // → { data: { user: { name: "Alice" } }, errors: [...] }
+```
+
+Access the actual data via `sourceData.stepId.data.data.user` (double `.data` — one for the envelope, one for the GraphQL response).
+
+### Pagination
+
+GraphQL APIs typically use cursor-based pagination (Relay-style). Use superglue's `cursorBased` pagination:
+
+```typescript
+{
+  body: '{"query": "query($cursor: String) { users(first: 50, after: $cursor) { edges { node { id name } } pageInfo { endCursor hasNextPage } } }", "variables": {"cursor": "<<cursor>>"}}',
+  pagination: {
+    type: "cursorBased",
+    pageSize: "50",
+    cursorPath: "data.users.pageInfo.endCursor",
+    stopCondition: "response.data.data.users.pageInfo.hasNextPage === false"
+  }
+}
+```
+
+Note the `cursorPath` and `stopCondition` must account for the GraphQL `data` wrapper.
+
+### Schema Discovery
+
+Superglue automatically introspects GraphQL schemas during documentation fetching. When a system URL contains "graphql", introspection is the first strategy tried — the full schema is fetched and stored as system documentation.
+
+Use `sg system search-docs` to query the stored schema for types, fields, and arguments before building tools. This is more reliable than manual introspection and works even when the API has introspection disabled for external callers.
+
+If the schema is not available via `sg system search-docs` (e.g. the system was created without a GraphQL URL, or introspection was blocked), test manually with `sg system call`:
+
+```json
+{ "query": "{ __typename }" }
+```
+
+If that succeeds, try a targeted introspection for the types you need:
+
+```json
+{ "query": "{ __type(name: \"User\") { fields { name type { name kind } } } }" }
+```
+
+## Common Pitfalls
+
+- Forgetting `Content-Type: application/json` header — most GraphQL servers reject requests without it
+- Accessing `sourceData.stepId.data.user` instead of `sourceData.stepId.data.data.user` — the GraphQL response has its own `data` wrapper inside the step envelope
+- Building complex variable objects inline with multiple `<<>>` expressions — use a transform step to build the variables object
+- Using `cursorPath` or `stopCondition` that doesn't account for the GraphQL `data` wrapper (e.g. `users.pageInfo` instead of `data.users.pageInfo`)
+- Not setting `modify: true` on mutation steps
+
+## Error Recovery
+
+When a GraphQL step fails:
+
+1. **Check GraphQL-level errors** — even 200 responses can contain `errors` in the response body. Check `sourceData.stepId.data.errors` for field-level or validation errors.
+2. **Verify the endpoint works** — use `sg system call` with `{ "query": "{ __typename }" }` to confirm connectivity and auth before debugging the query itself.
+3. **Check schema compatibility** — fields, types, and arguments change between API versions. Use `sg system search-docs` to check the auto-introspected schema first. If unavailable, use targeted `__type` queries or web search to verify the current schema.
+4. **Verify variables match the query signature** — GraphQL type mismatches (e.g. passing a string where `ID!` is expected) cause validation errors. Check that variable types in the query match what you're passing.
+5. **Check for auth scope issues** — some GraphQL APIs return partial data with `null` fields and errors when the token lacks required scopes for specific fields.

package/skills/superglue/references/http.md

@@ -0,0 +1,246 @@
+# HTTP
+
+## Step Configuration
+
+```typescript
+{
+  type: "request",
+  systemId: "my_api",
+  url: "<<my_api_url>>/v1/users",
+  method: "GET",
+  headers: { "Authorization": "Bearer <<my_api_access_token>>" },
+  queryParams: { "limit": "<<limit>>", "status": "active" },
+  body: '{"name": "<<(sourceData) => sourceData.currentItem.name>>"}',
+  pagination: { ... }
+}
+```
+
+All fields support `<<variable>>` and `<<(sourceData) => ...>>` expressions.
+
+### Fields
+
+| Field         | Required | Notes                                                   |
+| ------------- | -------- | ------------------------------------------------------- |
+| `url`         | yes      | Prefer `<<systemId_url>>/path` over hardcoded base URLs |
+| `method`      | yes      | `GET`, `POST`, `PUT`, `DELETE`, `PATCH`                 |
+| `headers`     | no       | Auth headers, content type, custom headers              |
+| `queryParams` | no       | Key-value pairs appended to URL                         |
+| `body`        | no       | Stripped for GET/HEAD/DELETE/OPTIONS                    |
+| `pagination`  | no       | See Pagination section                                  |
+| `systemId`    | no       | Links system credentials and URL. Omit for public APIs  |
+
+## Authentication
+
+Credentials are never injected automatically. You must place auth headers explicitly using `<<systemId_credentialKey>>` syntax.
+
+```javascript
+// Bearer token (including OAuth — token refresh is automatic, header is not)
+{ "Authorization": "Bearer <<systemId_access_token>>" }
+
+// API key
+{ "X-API-Key": "<<systemId_api_key>>" }
+
+// Basic Auth — auto-encoded to Base64, do NOT manually encode
+{ "Authorization": "Basic <<systemId_username>>:<<systemId_password>>" }
+
+// Runtime credentials from payload
+{ "Authorization": "Bearer <<(sourceData) => sourceData.user_access_token>>" }
+```
+
+### SOAP Authentication
+
+Some APIs (e.g. Salesforce) use SOAP XML login where credentials are embedded in the request body, not in headers. Multiple `<<>>` placeholders can appear adjacent — each is resolved independently and the results are concatenated.
+
+```typescript
+{
+  method: "POST",
+  url: "https://test.salesforce.com/services/Soap/u/59.0",
+  headers: {
+    "Content-Type": "text/xml; charset=UTF-8",
+    "SOAPAction": "login"
+  },
+  body: '<?xml version="1.0" encoding="utf-8"?><env:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"><env:Body><n1:login xmlns:n1="urn:partner.soap.sforce.com"><n1:username><<systemId_username>></n1:username><n1:password><<systemId_password>><<systemId_security_token>></n1:password></n1:login></env:Body></env:Envelope>'
+}
+```
+
+`<<systemId_password>><<systemId_security_token>>` resolves to the password and token concatenated — no separator needed. This works in URL, headers, and body fields.
+
+Headers starting with `x-` are treated as custom headers.
+Modern APIs expect auth in headers, not query parameters, unless docs explicitly say otherwise.
+
+## HTTP Runtime Details
+
+### Request Behavior
+
+- **GET, HEAD, DELETE, OPTIONS**: body is always stripped
+- **POST, PUT, PATCH**: body with leading `{` is JSON-parsed; empty body becomes undefined
+- Default headers: `Accept: */*`, Chrome-like `User-Agent`
+- HTTPS: `rejectUnauthorized: false` (accepts self-signed certs)
+
+### Body Resolution
+
+The runtime processes the body in this order:
+
+1. `<<>>` template variables are resolved — object/array returns are `JSON.stringify`'d before splicing
+2. The resulting string is JSON-parsed when possible
+3. `file::` references are resolved on the parsed object
+4. Final send format is determined:
+   - raw bytes when the entire body is `file::<key>.raw`
+   - base64 string for `file::<key>.base64`
+   - `multipart/form-data` when the `Content-Type` header requests it and body resolves to an object
+   - `JSON.stringify(...)` for objects
+   - plain string for everything else
+
+File references resolve only in the body, not in headers, query params, or URLs.
+
+**Double-encoding risk:** the runtime stringifies at both step 1 (expression into template) and step 4 (object into fetch body). When the body contains nested JSON (e.g. LLM APIs), have `<<>>` expressions return a string via `JSON.stringify(...)`, or use a preceding transform step.
+
+### File Uploads
+
+**Raw body** — send a file as the entire HTTP body:
+
+```typescript
+{
+  method: "PUT",
+  headers: { "Content-Type": "application/pdf" },
+  body: "file::document.raw"
+}
+```
+
+**Base64-in-JSON** — embed a base64-encoded file inside a structured payload:
+
+```typescript
+{
+  method: "POST",
+  headers: { "Content-Type": "application/json" },
+  body: '{"fileBase64":"file::document.base64"}'
+}
+```
+
+**Multipart form** — set the `Content-Type` header and provide a JSON object of form fields:
+
+```typescript
+{
+  method: "POST",
+  headers: { "Content-Type": "multipart/form-data" },
+  body: '{"file":"file::report.raw","description":"monthly export"}'
+}
+```
+
+The runtime builds a real `FormData` request: `file::...raw` values become file parts, scalars become text parts, non-file objects/arrays are JSON-stringified into text parts. The user-supplied `Content-Type` header is removed so `fetch` sets the boundary automatically.
+
+Multipart restrictions:
+
+- Body must resolve to a flat object of form fields
+- Nested file refs inside objects are not supported
+- Do not manually set a multipart boundary
+- Use `.raw` for file parts, `.base64` for base64 text fields, `.extracted` for parsed content fields
+
+### Response Handling
+
+All HTTP responses are read as raw bytes, then classified by byte-level detection (see `file-handling.md` for the full detection logic).
+
+- **Binary types** (PDF, Excel, DOCX, ZIP, GZIP): treated as files, parsed content also populates `data`. Later steps can reference via `file::<stepId>.raw/base64/extracted`.
+- **`application/octet-stream`**: always treated as a file.
+- **Responses over 25 MB**: treated as files as a safety fallback.
+- **RAW + `Content-Disposition: attachment`**: treated as a file.
+- **Structured/text responses** (JSON, XML, CSV, YAML, HTML, text under size limit): parsed inline to `data` only.
+
+### Error Detection
+
+Non-2xx responses throw with method, URL, response body preview (1000 chars), masked config, and retry count. Any response with status in the 2xx range is treated as success and returned to the step as-is — the runtime does not scan successful response bodies for error fields, so APIs that return HTTP 200 with `{ "error": "..." }` must be handled by your transform or by setting a `stopCondition`/assertion at the step level.
+
+Bypass: set `failureBehavior: "continue"` to skip the non-2xx throw and return the response anyway.
+
+### Retry Behavior
+
+**Connection/server errors:**
+
+- Default 5 retries, configurable via `retries` on the step
+- Retries on fetch-level errors (network/DNS/abort except timeout) and on 5xx responses
+- Fixed delay between retries (default 1000ms, configurable via `retryDelay`)
+
+**Rate limiting (429):**
+
+- Counts against the same retry budget as 5xx/network errors
+- Respects `Retry-After` header (seconds or HTTP-date)
+- Without header: exponential backoff `10^n * 1000ms + jitter` (capped at 1hr per wait)
+- Total wait across all 429 retries capped at `MAX_RATE_LIMIT_WAIT_MS` (1hr)
+
+**Timeouts:** Request timeout aborts the fetch and throws immediately — timeouts are not retried.
+
+### Pagination
+
+Makes multiple requests in a loop, merging results. Only configure after verifying the exact API pagination mechanism from docs.
+
+**Configuration:**
+
+```typescript
+pagination: {
+  type: "offsetBased" | "pageBased" | "cursorBased",
+  pageSize: "50",
+  cursorPath: "meta.next_cursor",   // cursorBased only — dot-path to cursor
+  stopCondition: "(response, pageInfo) => !response.data.meta.next_cursor"
+}
+```
+
+**Variables** — the matching variable MUST appear in the request (URL, queryParams, headers, or body). Throws if missing.
+
+| Type        | Variable                     | Starts at | Increments by            |
+| ----------- | ---------------------------- | --------- | ------------------------ |
+| pageBased   | `<<page>>`                   | 1         | 1                        |
+| offsetBased | `<<offset>>`                 | 0         | pageSize                 |
+| cursorBased | `<<cursor>>`                 | null      | extracted via cursorPath |
+| all         | `<<limit>>` / `<<pageSize>>` | pageSize  | —                        |
+
+**Stop conditions** — JS expression evaluated in sandbox. Receives `(response, pageInfo)`:
+
+- `response.data` = parsed API response body
+- `response.headers` = response headers
+- `pageInfo = { page, offset, cursor, totalFetched }`
+- Return `true` to STOP
+
+```javascript
+"!response.data.meta.next_cursor";
+"response.data.items.length === 0";
+"response.data.hasMore === false";
+"pageInfo.totalFetched >= 1000";
+```
+
+**Cursor extraction:** `cursorPath` is a simple dot-separated property path (e.g. `meta.next_cursor`, `data.0.id`) resolved against the parsed response body. It is not a JSONPath expression — wildcards, filters, and `$.` prefixes are not supported. For property names containing `@` (e.g. OData `@odata.nextLink`), use the name directly as the path. Null/undefined cursor stops pagination.
+
+**Safety checks (when stopCondition is set):**
+
+1. Identical pages 1 and 2 (both with data): throws — pagination params are not being applied
+2. Both pages empty + stop did not trigger: throws — broken stop condition
+3. Duplicate consecutive response after page 2: auto-stops
+
+**Without stopCondition (fallback):**
+
+- Array shorter than pageSize: stop
+- Duplicate response hash against any previously seen page: stop
+- Non-array response: stop after first request
+- Max 500 requests (vs 1000 max with a stopCondition)
+
+**Paginated step results:** all pages merge into a single result. Access via `sourceData.stepId.data` — it is NOT an array of per-page results. Do not call `.map()` on a paginated step result. Returns single object if one result, array if multiple (unwrapped from single-element array).
+
+## Common Pitfalls
+
+- Hardcoding base URLs that match the system's base URL instead of using `<<systemId_url>>`
+- Forgetting auth headers entirely — nothing is injected automatically, including for OAuth systems (only token refresh is automatic)
+- Putting pagination variables in the config but not setting a `pagination` block, or vice versa
+- Using `<<currentItem.id>>` — must use arrow function syntax: `<<(sourceData) => sourceData.currentItem.id>>`
+- Mixing multiple `<<>>` expressions in one body string when the API expects nested JSON — use a transform step instead
+- Setting `method: "POST"` for GraphQL queries without `modify: false` — POST that only reads should not be marked as modifying
+
+## Error Recovery
+
+When an HTTP step fails and the cause is not immediately clear:
+
+1. **Isolate endpoint vs system-level issues** — use `sg system call` to test a different endpoint on the same system. If multiple endpoints fail, the issue is likely global authentication or system configuration. If only one fails, the issue is endpoint-specific (wrong URL, missing params, incorrect method or authentication scopes).
+
+2. **Check credential scopes and system-side configuration** — a 403 or empty response often means the API key or OAuth token lacks required scopes, or the resource is restricted server-side. Use `sg system search-docs` or a web search to look up required scopes/permissions for the endpoint.
+
+3. **Verify request shape against API docs** — use `sg system search-docs` and web search to confirm the endpoint URL, required headers, query parameters, body format, and expected method. APIs change across versions. Search for the error message and the endpoint in docs and online.
+
+4. **Check for rate limiting** — 429 errors are retried automatically with backoff, but persistent 429s may mean the API key has hit a quota. Check API dashboard or docs for rate limit details.