@datarails-ai/dr-cli 0.0.1-locked → 1.6.70

package/dist/skills/connector-sandbox/SKILL.md

@@ -0,0 +1,95 @@
+---
+name: connector-sandbox
+description: Use when running or testing a Datarails connector locally in the sandbox / dev fix-and-test loop — "run fmp_fx locally", "test this connector against dev", "sandbox the connector", "why does the connector return no rows", "see the connector's output as a CSV" — before publishing. Wraps dr connectors dev-fetch + dev-run.sh and resolves the dev IDs for you.
+---
+
+# Connector Local Sandbox
+
+Run a connector's LOCAL working-tree code against REAL dev config + creds and see the vendor
+output as a local CSV — no publish→trigger cycle. Read-only: nothing lands in a filebox.
+
+Two commands: `dr connectors dev-fetch` (dr-cli — pulls dev config+creds into a run dir) →
+`scripts/dev-run.sh` (dr-connectors-runner — runs one hook in the flavor Docker image, writes
+`output.csv`). Full design/README: `dr-connectors-runner/README.md` → "Local fix-and-test loop".
+
+## Prerequisites (check first, in order)
+
+1. On the corporate **VPN** (the connectors control-plane is VPN-only).
+2. **Docker** running, and **AcrPull** works: `docker pull datarailsdev.azurecr.io/connectors-runner-base:latest`.
+3. `dr` logged in to the dev account that owns the connector (see resolution below).
+4. **Windows: run `dev-run.sh` under WSL2** (it needs Docker Desktop's `/mnt/c` bind mounts +
+   `host.docker.internal`). The `dr connectors dev-fetch` step runs fine in normal PowerShell.
+
+## Step 1 — Gather 3 inputs from the user
+
+Ask (via AskUserQuestion or plainly): **connector/vendor name** (e.g. `fmp_fx`), **server**
+(default `DEV`), **account** (if unsure, run `dr whoami --server DEV` — it lists stored accounts).
+Optional: `--hook` (default `fetch`), `--tag` (default `latest`). **Period (`--from/--to`):** required
+when the connector's query needs a period and the data-query stores none (check `.config.query` in
+step 2.4 — many connectors raise a hard error without a window); omit only if the connector doesn't
+use one.
+
+## Step 2 — Resolve the runtime IDs (REQUIRED SUB-SKILL: dr-cli). NEVER guess IDs from memory.
+
+The connector name is all you need; resolve the rest live. All reads — no confirmation needed.
+
+1. **Paths + account/org** — the connector lives at
+   `C:\codegithub\dr-connectors\dev\<orgId>_(<email>)\<vendor>\`. Glob for the `<vendor>` dir to get
+   `connections\<vendor>.py` + `dataqueries\<vendor>.py` (the `--connection`/`--query` paths) and the
+   org email. The dev account is usually `support@<email>` from that dir name (confirm in step 3).
+2. **Datasource id** — `dr integrations list --server <S> --account <A> --json`; pick the item whose
+   `name` contains the vendor and `type.name == "DatarailsConnector"` → its `id` (this is
+   `--datasource`).
+   - **Credentials:** `dev-fetch` passes the datasource's ENTIRE `config` through as the connector's
+     credentials — mirroring the production secret backend (`MonolithSecretBackend` returns
+     `ds["config"]` verbatim). So any auth shape works for any org: apikey, OAuth
+     `clientId`/`clientSecret`, service-account JSON, etc. — no per-field
+     allow-list. It only fails fast if the datasource has an empty/missing `config`.
+3. **Filebox + document id** — `dr filebox list --server <S> --account <A> --json`; find the
+   connector's target filebox → its `uuid` (the dev-fetch positional arg) and numeric `id` (this is
+   `--document-id`). Names can be ambiguous, so **confirm via step 4**.
+4. **Cross-check (disambiguates the filebox AND validates the wiring)** —
+   `dr filebox dq get <documentId> --server <S> --account <A> --json`: its `.connection` must equal
+   the datasource id from step 2, and `.config` holds `objectType`/`type`/`query`. If `.connection`
+   doesn't match, you picked the wrong filebox — try another.
+
+## Step 3 — Show resolved values, let the user confirm or override
+
+Print a table: account, datasource id, filebox uuid, document id, connection path, query path, hook,
+tag. Ask the user to confirm or override any value before running. Do NOT run until confirmed.
+
+## Step 4 — Run
+
+```bash
+# fetch (dr-cli; PowerShell is fine)
+dr connectors dev-fetch <fileboxUuid> --server <S> --account <A> \
+  --datasource <dsId> --document-id <docId> \
+  --connection "<path\to\connections\vendor.py>" --query "<path\to\dataqueries\vendor.py>" \
+  [--from <YYYY-MM-DD> --to <YYYY-MM-DD>] -o ./dev-run/<vendor>
+
+# run (dr-connectors-runner; WSL2 on Windows) — <run_dir> is the SAME dir as -o above
+./scripts/dev-run.sh <run_dir> --hook <hook> --tag <tag>
+```
+
+Expect `flavor=<f> image=… hook=<h>`, a `HookResult` verdict `ok`, and `<run_dir>/output.csv` with
+real rows. The fix-and-test loop: edit `connection.py`/`query.py`, re-run ONLY `dev-run.sh` (no
+re-fetch).
+
+## Interpreting failures
+
+| Symptom | Meaning |
+|---|---|
+| `verdict: retry`, error mentions a missing credential key (e.g. `'apikey'`, `'clientId'`) | `dev-fetch` forwards the whole datasource `config`, so the key the connector's `connect()` reads simply isn't stored on the datasource — inspect `dr integrations get <dsId> --show-secrets` and check it against what `connect()` expects. |
+| `datasource N has an empty/missing config` | The datasource has no `config` at all — nothing to use as credentials. Fix the datasource in the monolith first. |
+| `verdict: retry`, error mentions `'objectType'` | The data_query fields aren't under `config` — check `dr filebox dq get <docId> --json`. |
+| TLS / `self signed certificate` on `dr connectors …` | VPN is inspecting TLS — set `NODE_EXTRA_CA_CERTS=<gateway-ca.pem>`. Never `NODE_TLS_REJECT_UNAUTHORIZED=0`. |
+| `connection refused` to the creds stub | Not on WSL2 / `host.docker.internal` unreachable — see Prereq 4. |
+| `no such image` / manifest unknown | The flavor image tag isn't in ACR — try `--tag sha-<7digits>` of a deployed runner. |
+
+## Common mistakes
+
+- Guessing the filebox/datasource/document id (or trusting a memory note) instead of resolving live —
+  always run step 2 + the step-4 cross-check.
+- Running `dev-run.sh` from PowerShell on Windows (bind-mount/host.docker.internal breakage) — use WSL2.
+- Forgetting `--account` on a multi-identity server (the read commands error and list the options).
+- `secrets.json` in the run dir holds a real dev apikey — it's gitignored + chmod 600; never print or commit it.

package/dist/skills/datarails-connect-workato/SKILL.md

@@ -0,0 +1,140 @@
+---
+name: datarails-connect-workato
+description: Use when inspecting Datarails' Workato account from a terminal or script via the REST API — verifying the token, listing the parent workspace's recipes/connections, enumerating the 800+ embedded customer workspaces (managed_users), or drilling into one customer's connections/recipes/jobs to see what's syncing and why a recipe failed. Mentions of `WORKATO_API`, `wrkaus-`, `www.workato.com/api`, "workato recipe/connection", "managed user", "embedded customer workspace", "which customers are on workato", "why did the recipe fail", or a customer name whose bank/ERP feed runs on Workato all trigger this. READ-ONLY: only `GET` is permitted; the token is a live OEM admin key, so this skill never issues `POST`/`PUT`/`PATCH`/`DELETE`.
+---
+
+# Datarails Workato REST Access
+
+## Overview
+
+Datarails runs an **OEM / embedded** Workato account. The `WORKATO_API` token authenticates as the **parent workspace** ("Datarails Admin", `csintegrations@datarails.com`), which owns a handful of internal recipes **and** ~800 **managed_users** — one embedded Workato workspace per customer (their bank/ERP → Datarails feeds).
+
+| Concept | What it is |
+|---|---|
+| **Parent workspace** | The account the token belongs to. `GET /users/me`, `GET /recipes`, `GET /connections` operate here. |
+| **managed_user** | An embedded customer sub-workspace. Inspect via `GET /managed_users/{id}/...`. |
+
+## Auth
+
+| Field | Value |
+|---|---|
+| Base URL (US) | `https://www.workato.com/api` |
+| Base URL (EU) | `https://app.eu.workato.com/api` |
+| Header | `Authorization: Bearer $WORKATO_API` |
+| Env var | `WORKATO_API` — a Workato API-client token. Prefix **`wrkaus-`** = **US** datacenter (use the US base URL); a `wrkeu-`-style prefix would mean EU. |
+| Validate | `curl.exe -sS "https://www.workato.com/api/users/me" -H "Authorization: Bearer $env:WORKATO_API" -w "\nHTTP %{http_code}\n"` → `200` + account JSON. (Drop `-w` when piping into `jq` — the trailing `HTTP 200` text is not JSON and breaks the parse.) |
+
+`/users/me` returns the parent identity: `id`, `email`, `recipes_count`, `plan_id` (`oem_tbp_yearly`), billing period. Use `recipes_count` as the source of truth when paginating recipes.
+
+### Windows gotcha — `curl` is NOT curl (critical)
+
+In **Windows PowerShell 5.1, `curl` is an alias for `Invoke-WebRequest`**, which does not understand `-H`, `-sS`, or `-w`. **Always call `curl.exe`** explicitly, or use `Invoke-RestMethod` natively. The Bash tool has real `curl`.
+
+```powershell
+# PowerShell — real curl:
+curl.exe -sS "https://www.workato.com/api/users/me" -H "Authorization: Bearer $env:WORKATO_API"
+# or native, returns parsed objects:
+Invoke-RestMethod "https://www.workato.com/api/users/me" -Headers @{ Authorization = "Bearer $($env:WORKATO_API)" }
+```
+```bash
+# Bash tool:
+curl -sS "https://www.workato.com/api/users/me" -H "Authorization: Bearer $WORKATO_API"
+```
+
+### Transient 401 — retry before concluding the token is bad
+
+A **valid** token intermittently returns `401 Unauthorized` (empty body) and then `200` on the **identical** request seconds later. Before declaring the credential dead: confirm the env var is intact (length ~850, starts `wrkaus-`, no whitespace/newline) and **retry 2–3×**. A persistent 401 with a clean token = expired/revoked/wrong-datacenter.
+
+## Quick reference (verified endpoints)
+
+All `GET`. Auth header on every call. Base = `https://www.workato.com/api`.
+
+| Endpoint | Returns |
+|---|---|
+| `GET /users/me` | Parent identity: `id`, `email`, `recipes_count`, `plan_id`, billing period. |
+| `GET /recipes?per_page=100&page=N` | Parent recipes. Each: `id`, `name`, `running`, `job_succeeded_count`, `job_failed_count`, `last_run_at`, `applications`. |
+| `GET /connections?per_page=100` | Parent connections: `id`, `name`, `provider`, `authorization_status`. |
+| `GET /managed_users?per_page=100&page=N` | Embedded customer workspaces: `id`, `name`, `external_id`, `plan_id`. Paginate — there are ~800. |
+| `GET /managed_users/{id}` | One customer: `team_name`, `plan_id`, `task_count`, `active_connection_count`, `active_recipe_count`, `time_zone`, notification emails. |
+| `GET /managed_users/{id}/connections` | That customer's connections (same shape as parent; `.result` array). |
+| `GET /managed_users/{id}/recipes?per_page=100` | That customer's recipes (`.result` array). |
+| `GET /managed_users/{id}/properties?prefix=<p>` | That customer's **environment properties** as `{"result":{"<key>":"<value>"}}`. `prefix` is **required** (omitting it → HTTP 400 `Missing parameter prefix`); it's a starts-with match on the key, so `prefix=<Vendor>` returns every `<Vendor> *` property. **This is where recipe `account_property` pills resolve — including credentials** (API keys, account numbers, tokens a recipe reads via `_dp('{"pill_type":"account_property","property_name":"<key>"}')`). |
+
+### Environment properties = where recipe credentials live
+
+When a recipe's REST step sends a header like `Ocp-Apim-Subscription-Key: #{_dp('{"pill_type":"account_property","property_name":"<Vendor> API Key"}')}`, the value is **not** in the recipe code or the connection JSON — it's an **environment property** on that managed_user. Read it with the properties endpoint above (managed-user-scoped, `prefix` required):
+
+```bash
+WK="https://www.workato.com/api"; AUTH="Authorization: Bearer $WORKATO_API"; MU=<managed_user_id>
+curl -sS "$WK/managed_users/$MU/properties?prefix=<Vendor>" -H "$AUTH" \
+  | jq '.result'   # -> {"<Vendor> Account Number":"…","<Vendor> API Key":"…"}
+```
+
+- The `prefix` is the fastest way to enumerate a vendor's props — pass the property-name stem the recipe uses.
+- **The parent-scoped forms return `{}` — they do not work for a managed_user's props:** `GET /properties?prefix=<p>` (parent's own props only) and `GET /properties?managed_user_id={id}&prefix=<p>` (verified `{}`, `managed_user_id` is ignored). Always use the `/managed_users/{id}/properties` path.
+- Still `GET`, so read-only-safe — but these values are **live customer secrets**. Never echo, log, or commit them; pipe straight into the consuming command (e.g. build a datasource payload in the same shell call) and reference via a variable only. See the READ-ONLY / secrecy rules below.
+
+**List envelopes differ per endpoint** (verified) — never assume. Use the universal idiom **`(.items? // .result? // .)`** (the `?` matters: `.result` on a bare array is a *hard error*, not a catchable null):
+
+| Endpoint | Array lives under |
+|---|---|
+| `/recipes` | `.items` (`{count, items, page, per_page}`) |
+| `/connections` | **bare array** |
+| `/managed_users` | `.result` |
+| `/managed_users/{id}/recipes` | `.result` (`{count, page, per_page, result}`) |
+| `/managed_users/{id}/connections` | `.result` |
+
+Managed-user plan_ids seen: `oem_enterprise` (majority), `oem_enterprise_files`, `oem_enterprise_ai`, `datarails_plus`.
+
+**Endpoints that 404** (don't exist / wrong path): `GET /managed_users/{id}/usage`, `GET /managed_users/{id}/member_instances`.
+
+## Pagination — dedupe by `id`
+
+The `page` parameter is **1-based**; `page=0` can re-return page 1 and inflate counts with duplicates. Always **dedupe by `id`** and cross-check against `recipes_count` from `/users/me`. Stop when a page returns `< per_page` items.
+
+## Recipes helper (curl + jq)
+
+```bash
+WK="https://www.workato.com/api"; AUTH="Authorization: Bearer $WORKATO_API"
+
+# Parent recipes, one line each, most-run first (note .items envelope):
+curl -sS "$WK/recipes?per_page=100&page=1" -H "$AUTH" \
+  | jq -r '(.items? // .result? // .) | unique_by(.id) | sort_by(-.job_succeeded_count)[]
+           | "\(.id)\t\(if .running then "RUN" else "off" end)\t\(.job_succeeded_count)ok/\(.job_failed_count)fail\t\(.name)"'
+
+# Find a customer workspace by name:
+curl -sS "$WK/managed_users?per_page=100&page=1" -H "$AUTH" \
+  | jq -r '(.items? // .result? // .)[] | select(.name|test("Ares";"i")) | "\(.id)\t\(.plan_id)\t\(.name)"'
+
+# One customer's connections + recipes (swap in the id):
+ID=8109205
+curl -sS "$WK/managed_users/$ID"             -H "$AUTH" | jq '{team_name,plan_id,task_count,active_connection_count,active_recipe_count}'
+curl -sS "$WK/managed_users/$ID/connections" -H "$AUTH" | jq -r '(.items? // .result? // .)[] | "\(.name)\t\(.provider)\t\(.authorization_status // "UNAUTHORIZED")"'
+curl -sS "$WK/managed_users/$ID/recipes?per_page=100" -H "$AUTH" | jq -r '(.items? // .result? // .)[] | "\(if .running then "RUN" else "off" end)\t\(.job_succeeded_count)ok/\(.job_failed_count)fail\t\(.name)"'
+```
+
+`authorization_status`: `success` = wired; `null` = placeholder connection whose creds were never entered (common for the "ADD CLIENT ... CREDENTIALS" / SFTP stubs). A recipe with a nonzero `job_failed_count` is the first place to look when a customer's feed is stale.
+
+## Common mistakes
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `-H`/`-w` errors, HTML output in PowerShell | `curl` aliased to `Invoke-WebRequest` | Call `curl.exe`, or use `Invoke-RestMethod`. |
+| `401 Unauthorized`, empty body, token looks fine | Transient Workato auth hiccup | Retry 2–3×; a clean token usually succeeds on retry. |
+| `401` persists with a verified token | Expired/revoked or wrong datacenter | Regenerate in Workato admin → API clients; confirm `wrkaus-` → US base URL. |
+| Recipe count looks doubled (e.g. "231" when `/users/me` says 131) | `page=0` re-returned page 1 | Dedupe by `id`; page is 1-based; trust `recipes_count`. |
+| `404` on `/managed_users/{id}/usage` or `/member_instances` | Those paths don't exist | Use `/managed_users/{id}` for counts; `/connections` and `/recipes` for detail. |
+| `jq: Cannot iterate` | Endpoint returned a bare array, not `{result:[]}` | Handle both: `(.result // .)`. |
+
+## READ-ONLY — hard rule
+
+**This skill is strictly read-only. Only `GET` requests are permitted.** `WORKATO_API` is a **live OEM admin credential** with reach into 800+ customer workspaces, and any write is outward-facing — it can start/stop a customer's recipes, break a connection, or create/delete a managed_user, disrupting live financial data feeds.
+
+**Never issue `POST`, `PUT`, `PATCH`, or `DELETE` under this skill.** No exceptions:
+- Not "just to restart a failed recipe."
+- Not "just to re-authorize a connection."
+- Not because the user's phrasing implies a fix.
+
+If a task genuinely needs a write, **stop and surface it to the user as an explicit, separate request** — do not perform it as part of this skill. Describe what the write would do and let the user decide and run it deliberately.
+
+**Secrecy:** Never echo, commit, or paste the token into a web tool — reference it only as `$WORKATO_API`; when confirming it's set, log only length + first ~6 chars (`wrkaus`).