@superglue/cli 0.1.0

package/dist/skills/superglue/references/databases.md

@@ -0,0 +1,88 @@
+# Database Steps (PostgreSQL)
+
+## Step Configuration
+
+```typescript
+{
+  type: "request",
+  systemId: "my_postgres_db",
+  url: "postgres://<<my_postgres_db_user>>:<<my_postgres_db_password>>@<<my_postgres_db_host>>:<<my_postgres_db_port>>/<<my_postgres_db_database>>",
+  method: "POST",            // ignored for Postgres but required by schema
+  body: '{"query": "SELECT * FROM users WHERE id = $1", "params": [<<userId>>]}'
+}
+```
+
+Configure the connection url depending on which credentials are saved in the stored db system - reference them accordingly.
+
+## URL Format
+
+```
+postgres://user:password@host:port/database
+postgresql://user:password@host:port/database
+```
+
+All credential variables are resolved before connecting. Trailing slashes and extra slashes before query strings are cleaned.
+
+## Body Format
+
+JSON string with `query` and optional `params` (or `values`):
+
+```json
+{ "query": "SELECT * FROM users WHERE age > $1 AND status = $2", "params": [25, "active"] }
+```
+
+### Parameterized Queries
+
+**Always use parameterized queries** with `$1`, `$2`, etc. — prevents SQL injection.
+
+```json
+{ "query": "SELECT * FROM orders WHERE customer_id = $1", "params": ["<<customerId>>"] }
+```
+
+With expressions:
+
+```json
+{
+  "query": "INSERT INTO logs (message, level) VALUES ($1, $2)",
+  "params": ["<<(sourceData) => sourceData.message>>", "error"]
+}
+```
+
+With RETURNING:
+
+```json
+{
+  "query": "INSERT INTO users (name, email) VALUES ($1, $2) RETURNING *",
+  "params": [
+    "<<(sourceData) => sourceData.currentItem.name>>",
+    "<<(sourceData) => sourceData.currentItem.email>>"
+  ]
+}
+```
+
+## Connection Pooling
+
+- Pools cached by connection string, max 10 clients, 5s connection timeout
+- Idle pools cleaned up automatically
+- SSL auto-configured: enabled (with `rejectUnauthorized: false`) unless connecting to localhost without explicit sslmode
+
+## Retry Logic
+
+Default retries from server config. On final failure, error includes query text and params.
+
+## Return Value
+
+Returns `result.rows` — array of row objects with column names as keys.
+
+- `SELECT`: array of matching rows
+- `INSERT/UPDATE/DELETE` with `RETURNING`: array of returned rows
+- Mutations without `RETURNING`: empty array
+
+## Batch Operations via Loop
+
+Use a data selector returning an array to execute a query per item:
+
+```javascript
+// dataSelector: (sourceData) => sourceData.newUsers
+// body: {"query": "INSERT INTO users (name) VALUES ($1)", "params": ["<<(sourceData) => sourceData.currentItem.name>>"]}
+```

package/dist/skills/superglue/references/file-servers.md

@@ -0,0 +1,136 @@
+# File Server Steps (FTP / SFTP / SMB)
+
+FTP, SFTP, and SMB share an identical operation interface. Only the URL protocol and connection details differ.
+
+## URL Formats
+
+```
+ftp://user:password@host:port/basePath       (port 21)
+ftps://user:password@host:port/basePath      (port 21, TLS)
+sftp://user:password@host:port/basePath      (port 22)
+smb://user:password@host/sharename/basePath  (port 445)
+smb://domain\user:password@host/sharename/   (domain auth)
+```
+
+All support `<<variable>>` syntax in the URL.
+
+SMB requires a share name as the first path segment. Additional segments become a base path prepended to all operation paths.
+
+## Path Handling
+
+**All operation paths are relative to the base path in the URL.** Do NOT use absolute filesystem paths.
+
+If the system URL is `sftp://user:pass@host/home/sftptest/uploads`:
+
+- `report.csv` → `/home/sftptest/uploads/report.csv`
+- `subdir/data.csv` → `/home/sftptest/uploads/subdir/data.csv`
+
+Do NOT use `/home/sftptest/uploads/report.csv` — the base path is already set in the URL.
+
+If the URL has no base path (e.g., `sftp://user:pass@host`), operation paths are used as-is from the server root.
+
+## Body Format
+
+JSON string — single operation or array of operations (batch):
+
+### Single
+
+```json
+{ "operation": "get", "path": "data/report.csv" }
+```
+
+### Batch (sequential, same connection)
+
+```json
+[
+  { "operation": "mkdir", "path": "backup" },
+  { "operation": "get", "path": "data/report.csv" },
+  { "operation": "put", "path": "backup/report.csv", "content": "data here" }
+]
+```
+
+## Supported Operations
+
+| Operation | Required fields   | Returns                                                                 |
+| --------- | ----------------- | ----------------------------------------------------------------------- |
+| `list`    | `path`            | Array of `{ name, path, size, type, modifyTime, ... }`                  |
+| `get`     | `path`            | Auto-parsed file content (CSV→objects, JSON→parsed, etc.) or raw string |
+| `put`     | `path`, `content` | `{ success, message, size }`                                            |
+| `delete`  | `path`            | `{ success, message }`                                                  |
+| `rename`  | `path`, `newPath` | `{ success, message }`                                                  |
+| `mkdir`   | `path`            | `{ success, message }`                                                  |
+| `rmdir`   | `path`            | `{ success, message }`                                                  |
+| `exists`  | `path`            | `{ exists: boolean, path }`                                             |
+| `stat`    | `path`            | File metadata or `{ exists: false }`                                    |
+
+### `get` — Auto-Parsing
+
+Downloaded files are all parsed automatically:
+
+- CSV, JSON, XML, Excel, PDF, etc. → parsed to JS objects
+- Falls back to UTF-8 string
+- SMB additionally detects binary files (returns `{ _binary: true, encoding: "base64", data: "..." }`)
+
+### `put` — Content Handling
+
+- String → written as-is
+- Buffer → written directly
+- Object/Array → JSON.stringified with 2-space indent
+
+## Connection Details
+
+### SFTP
+
+Uses `ssh2-sftp-client`. Supports password + private key auth (`credentials.privateKey`, `credentials.passphrase`).
+
+### FTP/FTPS
+
+Uses `basic-ftp`. FTPS uses TLS with `rejectUnauthorized: false`.
+
+### SMB
+
+Uses `@awo00/smb2`. Connection lifecycle: create client → authenticate (domain optional) → connect tree (share) → execute → disconnect/logoff/close. Entire operation wrapped in timeout race.
+
+All paths use forward slashes (`/`) — SMB library handles Windows conversion internally.
+
+Credentials must be injected into the URL using placeholders:
+
+```
+smb://<<systemId_username>>:<<systemId_password>>@fileserver.example.com/ShareName
+```
+
+For private systems via Secure Gateway, the host is `tunnelId.tunnel` (e.g., `my_tunnel.tunnel`).
+
+## Return Value
+
+Single operation → result directly. Multiple operations → array of results.
+
+## Common Patterns
+
+### Download and process
+
+```json
+{ "operation": "get", "path": "incoming/<<(sourceData) => sourceData.currentItem.filename>>" }
+```
+
+### Upload results
+
+```json
+{
+  "operation": "put",
+  "path": "output/results.json",
+  "content": "<<(sourceData) => JSON.stringify(sourceData.processData.data)>>"
+}
+```
+
+### List then loop
+
+Step 1 lists files, step 2 uses data selector to loop over them:
+
+```javascript
+// Step 1: list files
+// body: {"operation": "list", "path": "incoming"}
+
+// Step 2 dataSelector: (sourceData) => sourceData.listFiles.data.filter(f => f.type === "file")
+// body: {"operation": "get", "path": "<<(sourceData) => sourceData.currentItem.path>>"}
+```

package/dist/skills/superglue/references/integration.md

@@ -0,0 +1,273 @@
+# Integrating superglue into Your Codebase
+
+After building and saving tools with the CLI, integrate them into your application.
+
+## Quick Reference: REST API (curl)
+
+For any language, use the REST API directly:
+
+```bash
+# Set your API key
+export SUPERGLUE_API_KEY="your-api-key"
+export SUPERGLUE_ENDPOINT="https://api.superglue.cloud"
+
+# Run a tool (sync - waits for completion)
+curl -X POST "$SUPERGLUE_ENDPOINT/v1/tools/my-tool-id/run" \
+  -H "Authorization: Bearer $SUPERGLUE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"inputs": {"userId": "123"}, "options": {"async": false}}'
+
+# Run a tool (async - returns immediately)
+curl -X POST "$SUPERGLUE_ENDPOINT/v1/tools/my-tool-id/run" \
+  -H "Authorization: Bearer $SUPERGLUE_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"inputs": {"userId": "123"}, "options": {"async": true}}'
+
+# Get run status/result
+curl "$SUPERGLUE_ENDPOINT/v1/runs/{runId}" \
+  -H "Authorization: Bearer $SUPERGLUE_API_KEY"
+
+# List tools
+curl "$SUPERGLUE_ENDPOINT/v1/tools?page=1&limit=50" \
+  -H "Authorization: Bearer $SUPERGLUE_API_KEY"
+
+# Get tool details
+curl "$SUPERGLUE_ENDPOINT/v1/tools/my-tool-id" \
+  -H "Authorization: Bearer $SUPERGLUE_API_KEY"
+
+# Cancel a running tool
+curl -X POST "$SUPERGLUE_ENDPOINT/v1/runs/{runId}/cancel" \
+  -H "Authorization: Bearer $SUPERGLUE_API_KEY"
+```
+
+### Response Format
+
+**Sync execution (200 OK):**
+
+```json
+{
+  "runId": "7f3e9c1a-2b4d-4e8f-9a3b-1c5d7e9f2a4b",
+  "toolId": "my-tool-id",
+  "status": "success",
+  "data": { "result": "..." },
+  "metadata": {
+    "startedAt": "2024-01-15T10:00:00Z",
+    "completedAt": "2024-01-15T10:00:05Z",
+    "durationMs": 5234
+  }
+}
+```
+
+**Async execution (202 Accepted):**
+
+```json
+{
+  "runId": "7f3e9c1a-2b4d-4e8f-9a3b-1c5d7e9f2a4b",
+  "toolId": "my-tool-id",
+  "status": "running"
+}
+```
+
+---
+
+## SDK Installation
+
+```bash
+# TypeScript/JavaScript
+npm install @superglue/client
+
+# Python
+pip install superglue-client
+```
+
+## TypeScript/JavaScript Usage
+
+```typescript
+import { configure, listTools, runTool, getRun } from "@superglue/client";
+
+// Configure once at startup
+configure({
+  apiKey: process.env.SUPERGLUE_API_KEY,
+  baseUrl: "https://api.superglue.cloud/v1", // or your self-hosted endpoint
+});
+
+// Run a tool (sync - waits for completion)
+const { data: result } = await runTool("my-tool-id", {
+  inputs: { userId: "123", date: "2024-01-15" },
+  options: { async: false },
+});
+
+if (result.status === "success") {
+  console.log(result.data);
+} else {
+  console.error(result.error);
+}
+```
+
+### Async Execution (Fire-and-Forget)
+
+```typescript
+// Run async - returns immediately
+const { data: run } = await runTool("my-tool-id", {
+  inputs: { userId: "123" },
+  options: { async: true },
+});
+
+console.log(`Run started: ${run.runId}`);
+
+// Poll for completion if needed
+let status = run;
+while (status.status === "running") {
+  await new Promise((r) => setTimeout(r, 1000));
+  const { data } = await getRun(status.runId);
+  status = data;
+}
+```
+
+### List and Discover Tools
+
+```typescript
+import { listTools, getTool, listSystems } from "@superglue/client";
+
+// List all tools
+const { data: tools } = await listTools({ page: 1, limit: 50 });
+console.log(tools.data);
+
+// Get tool details (including inputSchema)
+const { data: tool } = await getTool("my-tool-id");
+console.log(tool.inputSchema);
+
+// List systems
+const { data: systems } = await listSystems();
+```
+
+## Python Usage
+
+```python
+from superglue_client import SuperglueClient
+from superglue_client.api.tools import run_tool, list_tools, get_tool
+from superglue_client.api.runs import get_run
+
+# Create authenticated client
+client = SuperglueClient(
+    base_url="https://api.superglue.cloud/v1",
+    token=os.environ["SUPERGLUE_API_KEY"]
+)
+
+# Run a tool (sync)
+with client as c:
+    result = run_tool.sync(
+        client=c,
+        tool_id="my-tool-id",
+        body={"inputs": {"userId": "123"}, "options": {"async": False}}
+    )
+
+    if result.status == "success":
+        print(result.data)
+    else:
+        print(result.error)
+```
+
+### Async Python
+
+```python
+async with client as c:
+    result = await run_tool.asyncio(
+        client=c,
+        tool_id="my-tool-id",
+        body={"inputs": {"userId": "123"}}
+    )
+```
+
+## Webhook Triggers
+
+Trigger tools via HTTP POST without SDK. Webhooks execute asynchronously.
+
+```bash
+curl -X POST "https://api.superglue.cloud/v1/hooks/{toolId}?token={apiKey}" \
+  -H "Content-Type: application/json" \
+  -d '{"userId": "123"}'
+```
+
+### Webhook URL Pattern
+
+```
+POST {endpoint}/v1/hooks/{toolId}?token={apiKey}
+```
+
+- `endpoint` — Your superglue API endpoint
+- `toolId` — The saved tool ID (from `sg tool save`)
+- `apiKey` — Your superglue API key (passed as `token` query param)
+
+### Webhook Response
+
+Webhooks return `202 Accepted` immediately:
+
+```json
+{
+  "runId": "uuid-of-execution",
+  "status": "accepted",
+  "toolId": "my-tool-id"
+}
+```
+
+To get the result, poll the run status:
+
+```bash
+curl "https://api.superglue.cloud/v1/runs/{runId}" \
+  -H "Authorization: Bearer {apiKey}"
+```
+
+## Error Handling
+
+### Retry Pattern (TypeScript)
+
+```typescript
+async function runToolWithRetry(toolId: string, inputs: Record<string, any>, maxRetries = 3) {
+  for (let attempt = 1; attempt <= maxRetries; attempt++) {
+    const { data: result } = await runTool(toolId, { inputs, options: { async: false } });
+
+    if (result.status === "success") return result;
+
+    // Don't retry on validation errors
+    if (result.error?.includes("validation")) throw new Error(result.error);
+
+    // Exponential backoff for transient errors
+    if (attempt < maxRetries) {
+      await new Promise((r) => setTimeout(r, 1000 * Math.pow(2, attempt)));
+    }
+  }
+  throw new Error(`Tool ${toolId} failed after ${maxRetries} attempts`);
+}
+```
+
+### Fallback Pattern
+
+```typescript
+async function runWithFallback(inputs: Record<string, any>) {
+  const { data: primary } = await runTool("primary-tool", { inputs, options: { async: false } });
+  if (primary.status === "success") return primary;
+
+  console.warn("Primary tool failed, using fallback");
+  const { data: fallback } = await runTool("fallback-tool", { inputs, options: { async: false } });
+  return fallback;
+}
+```
+
+## Environment Configuration
+
+```bash
+# Required
+SUPERGLUE_API_KEY=your-api-key
+
+# Optional (defaults to hosted)
+SUPERGLUE_API_ENDPOINT=https://api.superglue.cloud/v1
+```
+
+## Best Practices
+
+1. **Store API keys in secrets** — Never commit to version control
+2. **Validate payloads client-side** — Check against `inputSchema` before calling
+3. **Handle errors gracefully** — Tools can fail due to external API issues
+4. **Monitor run history** — Use `sg run list` or the web UI to track executions
+5. **Use async mode for long-running tools** — Avoid timeouts on complex workflows

package/dist/skills/superglue/references/transforms-and-output.md

@@ -0,0 +1,92 @@
+# Transforms & Output Shaping
+
+Covers transform steps, output transforms, and response filters.
+
+## Three Transformation Points
+
+| Point                            | When it runs             | Purpose                              |
+| -------------------------------- | ------------------------ | ------------------------------------ |
+| `dataSelector`                   | Before step executes     | Controls input + single vs loop mode |
+| Transform step (`transformCode`) | As the step itself       | Reshape data between request steps   |
+| `outputTransform`                | After all steps complete | Shape final tool output              |
+
+## Transform Steps
+
+Use when you need intermediate data reshaping that can't wait until outputTransform (because a subsequent request step needs the data).
+
+```typescript
+{
+  id: "formatForInsert",
+  config: {
+    type: "transform",
+    transformCode: "(sourceData) => sourceData.getCustomers.data.map(c => ({ id: c.id, name: c.fullName }))"
+  }
+}
+```
+
+Transform steps do NOT have: systemId, url, method, headers, body, queryParams, or pagination.
+
+Results are wrapped in the **same envelope** as request steps:
+
+```javascript
+sourceData.formatForInsert = { currentItem: <dataSelector output>, data: <transformCode result>, success: true }
+```
+
+If the step has a dataSelector returning an array, the transform runs once per item.
+
+### When to Use
+
+- Data needed by a subsequent request step (can't defer to outputTransform)
+- Aggregating/combining results from multiple previous steps
+- Complex filtering or restructuring between API calls
+- Preparing data in a specific format for the next step
+
+### When NOT to Use
+
+- Simple input selection → use `dataSelector` instead
+- Final output shaping → use `outputTransform` instead
+- Simple filtering within a single step → use `dataSelector` returning a filtered array
+- As the final step before the outputTransform → merge into outputTransform code
+
+## Output Transform
+
+Final transformation shaping the tool's output. Runs after all steps complete.
+
+Access patterns depend on how the step was executed:
+
+- **Object-selector step**: `sourceData.stepId.data` — single envelope
+- **Array-selector (loop) step**: `sourceData.stepId.map(i => i.data)` — array of envelopes
+
+```javascript
+// Object-selector step — single result
+sourceData.getUsers.data; // the API response
+sourceData.getUsers.data.results; // nested field
+
+// Array-selector (loop) step — array of results
+sourceData.fetchDetails.map((i) => i.data); // all responses
+sourceData.fetchDetails.flatMap((i) => i.data.items); // nested arrays
+sourceData.fetchDetails.filter((i) => i.success).map((i) => i.data); // only successes
+```
+
+Payload fields are at root level — **NEVER** use `sourceData.payload.*`.
+
+### Requirements
+
+- Function signature: `(sourceData) => { ... }`
+- **Must have a return statement**
+- Pure synchronous — no async/await, no side effects
+- Handle missing data with `?.` and defaults
+- Validate arrays with `Array.isArray()` before array methods
+- Throw when expected required data is missing (enables self-healing)
+- NEVER include newlines or tabs in the code string
+
+## JS Code Constraints (all transform points)
+
+Runs in `isolated-vm` sandbox:
+
+- 4096 MB memory limit, 10s timeout (transforms/selectors), 3s timeout (pagination stops)
+- Synchronous only — no async/await, no Promises, no setTimeout
+- No side effects — no network, filesystem, or console access
+- No external dependencies — only standard JS built-ins (Array, Object, String, Math, Date, JSON, etc.)
+- JSON-serializable I/O only
+- Must return a value (undefined returns null)

package/package.json

@@ -0,0 +1,31 @@
+{
+  "name": "@superglue/cli",
+  "version": "0.1.0",
+  "bin": {
+    "sg": "./dist/cli.js"
+  },
+  "files": [
+    "dist/",
+    "skills/",
+    "skills.md"
+  ],
+  "dependencies": {
+    "commander": "^13.0.0",
+    "fast-json-patch": "^3.1.1"
+  },
+  "devDependencies": {
+    "@superglue/shared": "file:../shared",
+    "@types/node": "^20.0.0",
+    "tsup": "^8.0.0",
+    "typescript": "^5.0.0"
+  },
+  "scripts": {
+    "build": "tsup && cp -r skills dist/ 2>/dev/null || true",
+    "dev": "tsup --watch",
+    "type-check": "tsc --noEmit",
+    "prepublishOnly": "npm run build"
+  },
+  "publishConfig": {
+    "access": "public"
+  }
+}