@superglue/cli 1.1.17 → 1.1.19

package/dist/skills/superglue/references/http-apis.md

@@ -0,0 +1,248 @@
+# HTTP APIs
+
+Covers REST APIs, GraphQL, webhooks, and any HTTP-based service.
+
+## Step Configuration
+
+```typescript
+{
+  type: "request",
+  systemId: "my_api",
+  url: "<<my_api_url>>/v1/users",              // RECOMMENDED: use system URL variable for base URL
+  method: "GET",                               // GET, POST, PUT, DELETE, PATCH
+  headers: { "Authorization": "Bearer <<my_api_access_token>>" }, // supports <<variables>> and JS expressions
+  queryParams: { "limit": "<<limit>>", "status": "active" }, // supports <<variables>> and JS expressions
+  body: '{"name": "<<(sourceData) => sourceData.currentItem.name>>"}', // supports <<variables>> and JS expressions
+  pagination: { ... }  // optional
+}
+```
+
+### URL Best Practices
+
+**Prefer system URL variables over hardcoded URLs:**
+
+```typescript
+// RECOMMENDED: Environment-agnostic (works with dev/prod switching)
+{ "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account" }
+
+// ACCEPTABLE: When URL differs significantly from system base URL
+{ "url": "https://api.stripe.com/v1/customers" }
+
+// AVOID: Hardcoding URLs that match the system's base URL
+{ "url": "https://mycompany.salesforce.com/services/data/v58.0/sobjects/Account" }
+```
+
+Using `<<systemId_url>>` enables the same tool to work across different environments (dev/prod) without modification. The URL resolves dynamically based on execution mode.
+
+## Authentication Patterns
+
+```javascript
+// Bearer token
+{ "Authorization": "Bearer <<systemId_access_token>>" }
+
+// API key in header
+{ "X-API-Key": "<<systemId_api_key>>" }
+
+// Basic Auth — auto-encoded to Base64, do NOT manually encode
+{ "Authorization": "Basic <<systemId_username>>:<<systemId_password>>" }
+
+// Runtime credentials from payload
+{ "Authorization": "Bearer <<(sourceData) => sourceData.user_access_token>>" }
+```
+
+Headers starting with `x-` are treated as custom headers.
+Modern APIs expect auth in headers, NOT query parameters, unless docs explicitly say otherwise.
+
+## Request Behavior
+
+- **GET, HEAD, DELETE, OPTIONS**: Body is always stripped
+- **POST, PUT, PATCH**: Body with leading `{` is JSON-parsed; empty body becomes undefined
+- Default headers: `Accept: */*`, Chrome-like `User-Agent`
+- HTTPS: `rejectUnauthorized: false` (accepts self-signed certs)
+
+### Response Handling
+
+All HTTP responses are read as raw bytes, then classified by `detectFileType` (byte-level detection). See the file-handling skill for the full detection logic and file type reference.
+
+- **Binary types** (PDF, Excel, DOCX, ZIP, GZIP): treated as files. Parsed content also becomes `data`. Later steps can reference the file via `file::<stepId>.raw/base64/extracted`.
+- **`application/octet-stream`**: always treated as a file, even if byte-level detection falls back to `RAW`.
+- **Large responses over 25 MB**: treated as files as a safety fallback so the runtime does not try to parse a huge unknown blob as RAW text/JSON.
+- **RAW + Content-Disposition attachment**: treated as a file.
+- **Smaller structured/text responses** (JSON, XML, CSV, YAML, HTML, plain text under the size limit): parsed inline to `data` only.
+
+This means an HTTP step downloading a PDF or large `.bin` file will expose `stepFileKeys` and populate `sourceData.__files__`, while an ordinary JSON API response will usually just populate `sourceData.stepId.data`.
+
+### Body Resolution Order
+
+For HTTP request steps, the runtime processes the body in this order:
+
+1. `<<>>` template variables are resolved via `replaceVariables`
+2. The resulting body string is JSON-parsed when possible
+3. `file::...` references are resolved on the parsed body object
+4. The HTTP strategy decides how to send the final body:
+   - raw bytes for `file::<key>.raw` when the **entire body** is a file ref
+   - base64 string content for `file::<key>.base64` when the API expects base64 inside JSON/XML
+   - `multipart/form-data` when the `Content-Type` header requests it and the body resolves to an object
+   - plain string body for strings
+   - `JSON.stringify(...)` for ordinary objects
+
+File references are resolved **only in the body**, not in headers, query params, or URLs.
+
+**Avoiding double-encoding:** The runtime stringifies objects at both step 1 (expression → template) and step 4 (object → fetch body). When the body contains nested JSON (e.g. LLM APIs), have `<<>>` expressions return a string via `JSON.stringify(...)`, or use a preceding transform step.
+
+### File Uploads
+
+#### Raw Body Uploads
+
+If the endpoint expects the file as the entire HTTP body, use:
+
+```typescript
+{
+  method: "PUT",
+  headers: { "Content-Type": "application/pdf" },
+  body: "file::document.raw"
+}
+```
+
+This sends the exact original file bytes as the request body.
+
+#### Base64-In-JSON/XML Uploads
+
+If the endpoint expects the file encoded as base64 inside a structured payload, use:
+
+```typescript
+{
+  method: "POST",
+  headers: { "Content-Type": "application/json" },
+  body: '{"fileBase64":"file::document.base64"}'
+}
+```
+
+This resolves the file bytes to a base64 string before the final `JSON.stringify(...)`.
+
+#### Automatic Multipart Uploads
+
+If the endpoint expects `multipart/form-data`, set the header and make the body a JSON object of form fields:
+
+```typescript
+{
+  method: "POST",
+  headers: { "Content-Type": "multipart/form-data" },
+  body: '{"file":"file::report.raw","description":"monthly export"}'
+}
+```
+
+The runtime will:
+
+- build a real `FormData` request
+- turn `file::...raw` values into file parts using the stored filename and content type
+- turn scalar values into text parts
+- JSON-stringify non-file objects/arrays into text parts
+- remove the user-supplied `Content-Type` header and let `fetch` set the multipart boundary automatically
+
+Restrictions:
+
+- the multipart body must resolve to an object of top-level form fields
+- nested file refs inside objects are not supported
+- do **not** manually include a multipart boundary in the header
+- use `.raw` for file parts, `.base64` for base64 text fields, and `.extracted` only for text fields that should contain parsed content
+
+## Error Detection
+
+### HTTP Status Errors
+
+Non-2xx responses throw with: method, URL, response body preview (1000 chars), masked config, retry count.
+
+### Smart Error Detection in 2xx Responses
+
+Even successful responses are scanned for error indicators:
+
+- `response.code` or `response.status` is 400-599 → throws
+- Keys matching `error`, `errors`, `error_message`, `failure_reason`, `failure`, `failed` (up to depth 2) with non-empty values → throws
+- Error message includes: _"To prevent this from happening, enable 'Continue on failure' in the step's Advanced Settings."_
+
+Bypass: `failureBehavior: "continue"` skips all error checking. Can be set in the tool playground.
+
+## Retry Logic
+
+### Connection/Server Errors
+
+- Default 1 retry (3 if keep-alive disabled), capped at server max
+- Only retries if response was fast (under quick-retry threshold)
+- Configurable delay between retries
+
+### Rate Limiting (429)
+
+- Respects `Retry-After` header (seconds or date)
+- Without header: exponential backoff `10^n * 1000ms + jitter` (max 1hr per wait)
+- Total wait capped at `MAX_RATE_LIMIT_WAIT_MS`
+- Separate from general retry count
+
+## Pagination
+
+Makes multiple requests in a loop, merging results. Only configure if you've verified the exact API pagination mechanism from docs.
+
+### Configuration
+
+```typescript
+pagination: {
+  type: "offsetBased" | "pageBased" | "cursorBased",
+  pageSize: "50",
+  cursorPath: "meta.next_cursor",   // cursorBased only — JSONPath to cursor
+  stopCondition: "(response, pageInfo) => !response.data.meta.next_cursor"
+}
+```
+
+### Variables (auto-injected)
+
+| Type        | Variable                     | Starts at | Increments by            |
+| ----------- | ---------------------------- | --------- | ------------------------ |
+| pageBased   | `<<page>>`                   | 1         | 1                        |
+| offsetBased | `<<offset>>`                 | 0         | pageSize                 |
+| cursorBased | `<<cursor>>`                 | null      | extracted via cursorPath |
+| all         | `<<limit>>` / `<<pageSize>>` | pageSize  | —                        |
+
+**CRITICAL**: The matching pagination variable MUST appear in the request (URL, queryParams, headers, or body). Throws if missing.
+
+### Stop Conditions
+
+JS expression evaluated in sandbox. Receives `(response, pageInfo)`:
+
+- `response.data` = parsed API response body
+- `response.headers` = response headers
+- `pageInfo = { page, offset, cursor, totalFetched }`
+- Return `true` to **STOP**
+
+Examples:
+
+```javascript
+"!response.data.meta.next_cursor"; // no next cursor
+"response.data.items.length === 0"; // empty page
+"response.data.hasMore === false"; // explicit flag
+"pageInfo.totalFetched >= 1000"; // item cap
+```
+
+### Safety Checks (only when stopCondition is set)
+
+1. **Identical pages 1 & 2** (both with data): Throws — pagination params aren't being applied
+2. **Both pages empty + stop didn't trigger**: Throws — broken stop condition
+3. **Duplicate consecutive response (after page 2)**: Auto-stops
+
+### Without stopCondition (fallback)
+
+- Array shorter than pageSize → stop
+- Duplicate response hash (against any previously seen page) → stop
+- Non-array response → stop after first request
+- Max 500 requests (vs 1000 max with a stopCondition)
+
+### Cursor Extraction
+
+`cursorPath` is a JSONPath expression (auto-prefixed with `$.` if needed). Extracted via `jsonpath-plus`. Null cursor → stop.
+
+**Special characters in cursorPath:** For property names containing `@` (e.g. OData's `@odata.nextLink`), use the property name directly as the cursorPath: `cursorPath: "@odata.nextLink"`. The engine handles `@`-prefixed paths automatically.
+
+## Output
+
+Paginated steps merge all pages into a **single** result object. The step result is always accessed via `sourceData.stepId.data` — it is NOT an array of per-page results. Do NOT call `.map()` on a paginated step result.
+
+Returns single object if one result, array if multiple (unwrapped from single-element array).

package/dist/skills/superglue/references/redis.md

@@ -7,7 +7,6 @@
   type: "request",
   systemId: "my_redis",
   url: "redis://<<my_redis_username>>:<<my_redis_password>>@<<my_redis_host>>:<<my_redis_port>>/<<my_redis_database>>",
-  method: "POST",            // ignored for Redis but required by schema
   body: '{"command": "GET", "args": ["<<keyName>>"]}'
 }
 ```
@@ -23,7 +22,7 @@ rediss://user:password@host:port/database   (TLS)
 
 - `redis://` for standard connections
 - `rediss://` for TLS-encrypted connections (e.g. Redis Cloud, AWS ElastiCache with encryption)
-- `/database` is the database number (0-15), defaults to 0 if omitted
+- `/database` is the database number (defaults to 0 if omitted; max depends on the server's `databases` config, commonly 16)
 
 All credential variables are resolved before connecting. Trailing slashes are cleaned.
 
@@ -120,7 +119,7 @@ Multiple commands:
 ## Connection Management
 
 - A fresh connection is created per execution and closed after completion
-- TLS auto-configured for `rediss://` URLs (with `rejectUnauthorized: false`)
+- TLS auto-configured for `rediss://` URLs (currently uses `rejectUnauthorized: false`)
 - 5-second connection timeout, 30-second command timeout
 - Array commands (pipeline) run on a single connection for efficiency
 
@@ -133,7 +132,7 @@ Default retries from server config. On final failure, error includes command and
 Returns the raw Redis response:
 
 - `GET` / `HGET`: string or null
-- `HGETALL`: object with field-value pairs (flattened array from Redis, parsed by ioredis)
+- `HGETALL`: flat array of alternating field-value pairs (e.g., `["name", "Alice", "email", "alice@example.com"]`)
 - `LRANGE` / `SMEMBERS` / `KEYS`: array of strings
 - `SET` / `DEL` / `EXPIRE`: "OK" or integer
 - `INCR` / `LLEN` / `SADD`: integer
@@ -186,5 +185,5 @@ Use a data selector returning an array to execute a command per item:
 
 - All args must be strings (Redis protocol is text-based) - numbers are auto-coerced
 - Use `SCAN` instead of `KEYS` in production for large keyspaces
-- `HGETALL` returns an empty object `{}` for non-existent keys
+- `HGETALL` returns an empty array `[]` for non-existent keys
 - Use array body for multi-command pipelines instead of separate steps

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@superglue/cli",
-  "version": "1.1.17",
+  "version": "1.1.19",
   "bin": {
     "sg": "./dist/cli.js"
   },
@@ -15,6 +15,7 @@
     "fast-json-patch": "3.1.1"
   },
   "devDependencies": {
+    "@superglue/shared": "file:./_shared",
     "@types/node": "25.5.0",
     "tsup": "8.5.1",
     "typescript": "5.8.3"

package/skills/superglue/SKILL.md

@@ -19,8 +19,9 @@ description: "Build, test, deploy and integrate superglue tools via the sg CLI a
 
 **Authentication & Credentials:**
 
-- **Auth headers MUST be set explicitly** — credentials are NEVER auto-injected into requests or tool steps
-- Every request step (in tools AND `sg system call`) must include the appropriate auth header using `<<systemId_credentialKey>>` syntax
+- **Authentication must always be explicitly configured** — nothing is injected automatically in any protocol
+- For HTTP: include auth headers using `<<systemId_credentialKey>>` syntax (e.g., `"Authorization": "Bearer <<systemId_access_token>>"`)
+- For databases/Redis/file servers: embed credential placeholders in the connection URL (e.g., `postgres://<<sys_user>>:<<sys_pass>>@host/db`)
 - OAuth systems also require explicit headers — only token refresh is automatic
 
 **Saving Tools:**
@@ -31,7 +32,7 @@ description: "Build, test, deploy and integrate superglue tools via the sg CLI a
 
 **Credentials in Non-Interactive Environments (AI Agents, CI):**
 
-- **Pass ALL credentials (including secrets) directly via `--credentials` JSON** — do NOT use `--sensitive-credentials` (it requires an interactive TTY which agents don't have)
+- **Pass ALL credentials (including secrets) directly via `--credentials` JSON**
 - Ask the user to provide secret values, then pass them in `--credentials '{"api_key":"sk-xxx"}'`
 
 ---
@@ -121,15 +122,67 @@ sg tool save --draft <draftId>
 
 | Flag               | Description                                 |
 | ------------------ | ------------------------------------------- |
-| `--json`           | Force JSON output (auto-enabled in non-TTY) |
+| `--table`          | Human-readable table output (default: JSON) |
+| `--full`           | Disable truncation of large fields          |
 | `--api-key <key>`  | Override API key                            |
 | `--endpoint <url>` | Override API endpoint                       |
 
+## Capability Presets
+
+The CLI supports three capability presets that control which commands are available. Set during `sg init` or via `SUPERGLUE_CLI_PRESET` env var. Stored in `config.json` as `"preset"`.
+
+| Preset    | Description                                                        |
+| --------- | ------------------------------------------------------------------ |
+| `runner`  | Run saved tools by ID only. Read-only lookups. No building/editing |
+| `builder` | Runner + build/edit/save tools, call systems. No system CRUD       |
+| `admin`   | Full access (default)                                              |
+
+### Preset-to-Command Matrix
+
+```
+Command                  runner   builder   admin
+─────────────────────────────────────────────────
+sg init                    x         x        x
+sg update                  x         x        x
+sg skill                   x         x        x
+sg tool build              -         x        x
+sg tool run --tool         x         x        x
+sg tool run --config/etc   -         x        x
+sg tool edit               -         x        x
+sg tool save               -         x        x
+sg tool find / list        x         x        x
+sg system create           -         -        x
+sg system edit             -         -        x
+sg system oauth            -         -        x
+sg system call             -         x        x
+sg system search-docs      -         x        x
+sg system find / list      x         x        x
+sg run list                x         x        x
+sg run get                 x         x        x
+```
+
+### Setting the Preset
+
+```bash
+# During init (interactive or non-interactive)
+sg init --preset runner --api-key "..." --global
+
+# Via environment variable (overrides config.json)
+export SUPERGLUE_CLI_PRESET=builder
+
+# Manually edit .superglue/config.json
+{ "preset": "runner", "apiKey": "...", ... }
+```
+
+Blocked commands print a clear error with the current preset name and how to change it.
+
 ## Command Reference
 
 ### `sg init`
 
-Interactive setup — API key, endpoint, output mode. Creates `.superglue/`.
+Interactive setup — API key, endpoint, output mode, capability preset. Creates `.superglue/`.
+
+Non-interactive: `sg init --api-key "..." --preset runner --global`
 
 ### Tool Commands
 
@@ -212,17 +265,16 @@ sg system create --name "Slack" --template slack
 
 **Full flag reference:**
 
-| Flag                               | Description                                                                  |
-| ---------------------------------- | ---------------------------------------------------------------------------- |
-| `--name <name>`                    | Human-readable name — **REQUIRED**                                           |
-| `--id <id>`                        | System ID (lowercase, underscores only) — derived from name if omitted       |
-| `--url <url>`                      | API base URL — **REQUIRED** (auto-filled by `--template`)                    |
-| `--template <id>`                  | Template ID (stripe, shopify, slack, etc.)                                   |
-| `--credentials <json>`             | Credentials JSON (use for ALL creds in non-interactive mode)                 |
-| `--sensitive-credentials <fields>` | Comma-separated fields for masked input (TTY only — do NOT use in AI agents) |
-| `--docs-url <url>`                 | Documentation URL to scrape                                                  |
-| `--openapi-url <url>`              | OpenAPI spec URL                                                             |
-| `--env <environment>`              | Environment: `dev` or `prod` (default: prod)                                 |
+| Flag                   | Description                                                            |
+| ---------------------- | ---------------------------------------------------------------------- |
+| `--name <name>`        | Human-readable name — **REQUIRED**                                     |
+| `--id <id>`            | System ID (lowercase, underscores only) — derived from name if omitted |
+| `--url <url>`          | API base URL — **REQUIRED** (auto-filled by `--template`)              |
+| `--template <id>`      | Template ID (stripe, shopify, slack, etc.)                             |
+| `--credentials <json>` | Credentials JSON (all fields including secrets)                        |
+| `--docs-url <url>`     | Documentation URL to scrape                                            |
+| `--openapi-url <url>`  | OpenAPI spec URL                                                       |
+| `--env <environment>`  | Environment: `dev` or `prod` (default: prod)                           |
 
 **`sg system edit`** — Update system config or credentials. Supports `--env dev|prod`.
 **`sg system list` / `sg system find`** — List or search systems. `find` supports `--env dev|prod`.
@@ -367,7 +419,7 @@ Basic auth is auto-encoded: if the value after `Basic ` isn't already base64, th
 ### Credential Lifecycle
 
 - Non-sensitive creds (`client_id`, `auth_url`, etc.) are stored via `--credentials`
-- Sensitive creds (`api_key`, `client_secret`) — in non-interactive mode (AI agents, CI), pass via `--credentials` JSON. In interactive TTY, use `--sensitive-credentials` for masked input, or set `SUPERGLUE_CRED_<FIELD>` env vars.
+- Sensitive creds (`api_key`, `client_secret`) — pass via `--credentials` JSON: `--credentials '{"client_secret":"..."}'`
 - OAuth tokens (`access_token`, `refresh_token`) are auto-refreshed before each step execution
 
 ---
@@ -549,9 +601,8 @@ All types stop when `stopCondition` returns true, response is empty, or max page
 ### Credential Model
 
 - `credentials` — all credentials passed as JSON via `--credentials`
-- `sensitiveCredentials` — interactive masked prompt for secrets (TTY only, NOT for AI agents or CI)
-- **For AI agents / non-interactive use:** pass ALL credentials (including secrets like api_key, client_secret) directly via `--credentials '{"api_key":"..."}'`. Ask the user to provide the secret values, then include them in the JSON.
-- **For human interactive use:** use `--sensitive-credentials` for masked input of secrets
+- **Pass ALL credentials** (including secrets like api_key, client_secret) directly via `--credentials '{"api_key":"..."}'`. Ask the user to provide the secret values, then include them in the JSON.
+- OAuth tokens (`access_token`, `refresh_token`) are auto-refreshed before each step execution
 
 ### Templates
 
@@ -594,6 +645,165 @@ sg system oauth --system-id gmail --scopes "https://www.googleapis.com/auth/gmai
 
 ---
 
+## Tool Building Guide
+
+### Build Recipe
+
+Before producing a tool config:
+
+1. Load relevant skills: data-handling and protocol skill(s) for the involved systems
+2. Use `sg system find` for every involved system — note `storedCredentials` and URL
+3. Use `sg system search-docs` for each system — look up endpoints, auth patterns, pagination, response structure
+4. Use `sg system call` to test 1-2 primary endpoints — verify response structure and field names before building
+5. Only then call `sg tool build --config '...'` with the full tool config
+
+### Planning Steps
+
+- Fetch prerequisites: available projects, entity types, categories, etc.
+- Each step = one API call, one DB query, one file operation, or a transform step (no compound ops)
+- Final aggregation/filtering/sorting should happen in the `outputTransform`, not in a step
+- Step instructions: 2-3 sentences describing the goal and expected data
+- `instruction`: Write a 1-2 sentence summary of the tool's purpose. Never leave empty.
+
+### Step Result Envelopes
+
+Every step result is wrapped — you MUST account for this in dataSelectors and outputTransform.
+
+- **Object selector (or none)** — access via `sourceData.stepId.data`
+- **Array selector** — access via `sourceData.stepId.map(i => i.data)`
+- **Paginated step** — access via `sourceData.stepId.data` (pages are merged server-side into a single envelope — do NOT `.map()` over it)
+
+NEVER access step results without `.data` — `sourceData.stepId.results` will fail because you're hitting the envelope, not the API response.
+
+### modify Flag
+
+Set `modify: true` only when the step writes, updates, or deletes live data. Don't rely on HTTP method alone — a POST that only reads (e.g., GraphQL query) should be `modify: false`. Default is false.
+
+### Validation Rules
+
+- Tool must have a valid `id` string
+- Tool must have a `steps` array
+- `systemId` is optional on request steps. Setting it makes that system's credentials available as `<<systemId_credKey>>` template variables, and enables `<<systemId_url>>` for the base URL. Omit for public APIs.
+- Every request step must have a non-empty `url`
+- Transform steps must have `transformCode`
+
+### Key Pitfalls
+
+- NEVER guess API endpoints — always verify with `sg system search-docs` or `sg system call` first
+- NEVER put `systemId` on the step object — it belongs inside `step.config.systemId`
+- NEVER use `<<(sourceData) => sourceData.payload.X>>` — payload fields are at root level of sourceData
+- NEVER use `<<currentItem.id>>` — use arrow function syntax: `<<(sourceData) => sourceData.currentItem.id>>`
+- NEVER hardcode pagination params — use `<<page>>`, `<<offset>>`, `<<cursor>>`, `<<limit>>`
+- NEVER add an `outputSchema` unless the user explicitly requested a specific response structure
+- NEVER leave instructions empty
+- ALWAYS explicitly configure authentication in every step — credentials are never automatically included in any protocol
+- PREFER system URL variables (`<<systemId_url>>`) over hardcoded base URLs
+- Check `sg system find` output for `storedCredentials` to know the exact variable names available
+- `outputTransform` must be a single-line JS string (no literal newlines or tabs)
+- NEVER use regex literals with `/` or complex escapes in transforms — they corrupt during serialization. Use `new URL()`, `.split()`, or `new RegExp()` instead. See data-handling reference "Serialization Safety".
+- For complex request bodies, use a preceding transform step — don't put multi-statement logic inside `<<>>` expressions
+- When the body contains nested/stringified JSON (e.g. LLM APIs): have `<<>>` expressions return a string via `JSON.stringify(...)`, and use a single expression for the whole body rather than mixing `<<>>` expressions with static JSON
+
+### Complex Body Construction
+
+When a request body needs data from multiple steps or requires aggregation, use a preceding transform step:
+
+```javascript
+// Step: prepareBody (transform)
+transformCode: "(sourceData) => { var items = sourceData.step1.data.results; return { data: items.filter(function(i) { return i.active; }), count: items.length }; }";
+
+// Step: submitData (request)
+body: "<<(sourceData) => JSON.stringify(sourceData.prepareBody.data)>>";
+```
+
+---
+
+## Tool Editing Guide
+
+Edit tools using RFC 6902 JSON Patch operations via `sg tool edit`.
+
+### Patch Format
+
+Each operation:
+
+```typescript
+{
+  op: "add" | "remove" | "replace" | "move" | "copy" | "test",
+  path: string,      // JSON Pointer (e.g., "/steps/0/config/url")
+  value?: any,       // required for add, replace, test
+  from?: string      // required for move, copy
+}
+```
+
+### Key Paths
+
+- `/instruction` — tool description
+- `/steps/N` — step at index N
+- `/steps/N/config/url`, `/steps/N/config/method`, `/steps/N/config/headers`, etc.
+- `/steps/N/dataSelector` — JS function controlling input + loop mode
+- `/outputTransform` — final output shaping function
+- `/outputSchema` — optional JSON schema for output
+
+### Operations
+
+- **`replace`** — Change an existing value. **Path must exist** — if the field is `null` or absent, `replace` fails. Use `add` instead for optional fields.
+- **`add`** — Creates the field if missing, overwrites if it exists. **Prefer `add` over `replace`** when the field might be null or absent. Append to array with `/steps/-`.
+- **`remove`** — Delete field or array element.
+- **`move`** — Reorder or relocate.
+
+### Validation Rules
+
+- Never patch `id` — tool IDs and draft IDs are immutable when editing
+- `op` and `path` are required on every patch; `path` must start with `/`
+- Patches are applied **sequentially** — later patches see effects of earlier ones
+- After patches, same validation rules as building apply (valid id, steps array, URLs present)
+
+### Principles
+
+- **Check result wrapping** — before writing any dataSelector or outputTransform patch, determine whether the upstream step used an object or array selector, and access `.data` accordingly
+- **Don't forget downstream** — if you change a step's output shape, check if later dataSelectors and outputTransform need updating
+- **Index shifts** — when removing array elements, subsequent indices shift down. Account for this in multi-patch operations.
+
+---
+
+## Systems Handling Guide
+
+Systems are reusable building blocks that store configuration details and credentials.
+
+### Credential Use in Tool Configurations
+
+For every `systemId` referenced in a tool, credentials are namespaced as `<<systemId_credentialKey>>` and must be explicitly placed in the step config. For HTTP: in headers. For databases/Redis/file servers: in the connection URL. Nothing is injected automatically.
+
+```
+System id="stripe", storedCredentials: api_key → "sk_proj****"
+→ Available as: <<stripe_api_key>>
+```
+
+OAuth tokens (`access_token`, `refresh_token`) are auto-refreshed before each step execution. Access tokens must also be explicitly referenced in step headers.
+
+### System URL Variables
+
+System URLs are available as `<<systemId_url>>` and should be used instead of hardcoding base URLs:
+
+```
+System id="salesforce", url="https://mycompany.salesforce.com"
+→ Available as: <<salesforce_url>>
+
+Step config: { "url": "<<salesforce_url>>/services/data/v58.0/sobjects/Account" }
+```
+
+### Dev/Prod Environments
+
+`environment` is `"dev"` or `"prod"` (or unset for legacy systems). Set at creation only — immutable afterward. Two systems with the same ID but different environments are automatically linked. Dev systems inherit documentation from their linked prod system. When creating a dev system, always ask for new credentials — never copy from prod.
+
+Tools reference systems by ID only. Execution mode determines which environment is used (dev mode falls back to prod if no dev system exists).
+
+### System-Specific Instructions
+
+Systems may include `specificInstructions` from the user (visible in `sg system find` output). Follow them when present. These capture API constraints, rate limits, special endpoints, auth requirements, or usage patterns.
+
+---
+
 ## Debugging Failed Tools
 
 ### Debug Workflow
@@ -653,13 +863,16 @@ with client as c:
 
 For detailed documentation on specific topics, read these files in the `references/` directory:
 
-| File                                  | When to read                                                                                     |
-| ------------------------------------- | ------------------------------------------------------------------------------------------------ |
-| `references/integration.md`           | **READ THIS** when deploying tools to production - SDK usage, REST API, webhooks, error handling |
-| `references/databases.md`             | Building tools that query PostgreSQL or Microsoft SQL Server (Azure SQL) databases               |
-| `references/redis.md`                 | Building tools that interact with Redis (commands, pipelines, data types)                        |
-| `references/file-servers.md`          | Building tools that interact with FTP/SFTP/SMB file servers                                      |
-| `references/transforms-and-output.md` | Complex data transformations, output shaping, JS sandbox constraints                             |
+| File                          | When to read                                                                                     |
+| ----------------------------- | ------------------------------------------------------------------------------------------------ |
+| `references/integration.md`   | **READ THIS** when deploying tools to production - SDK usage, REST API, webhooks, error handling |
+| `references/databases.md`     | Building tools that query PostgreSQL or Microsoft SQL Server (Azure SQL) databases               |
+| `references/redis.md`         | Building tools that interact with Redis (commands, pipelines, data types)                        |
+| `references/file-servers.md`  | Building tools that interact with FTP/SFTP/SMB file servers                                      |
+| `references/data-handling.md` | Variables, selectors, result envelopes, transforms, JS sandbox constraints                       |
+| `references/file-handling.md` | File detection, parsing, producedFiles, file references, aliasing rules                          |
+| `references/http-apis.md`     | HTTP step config — auth patterns, pagination, retries, error detection                           |
+| `references/access-rules.md`  | RBAC roles, tool/system permissions, custom rules, mutation detection (enterprise only)          |
 
 **Important:** When the user asks about integrating superglue into their codebase, deploying tools, using the SDK, or calling tools from code, you MUST read `references/integration.md` for complete examples.