@codemation/agent-skills 0.4.0 → 0.5.2

package/skills/builder/workflow-dsl/SKILL.md

@@ -0,0 +1,492 @@
+---
+name: workflow-dsl
+description: Write a Codemation workflow definition with the fluent builder (createWorkflowBuilder().trigger().then().when().build()) and the built-in node classes from @codemation/core-nodes. Use this first whenever authoring or editing any workflow under src/workflows.
+compatibility: Designed for Codemation apps and plugins that author workflows.
+tags: workflow, dsl, authoring
+uses: "@codemation/core-nodes, @codemation/core"
+---
+
+# Codemation Workflow DSL
+
+A workflow is a `default`-exported definition: a **trigger** that emits items, then **node steps** that transform them. Items are `{ json, binary? }`; each node runs **per item** in a batch. You build with the fluent chain `createWorkflowBuilder({ id, name }).trigger(...).then(...).build()` and the node classes below — that is the whole surface. The per-item type flows through `.then(...)` via generics, so annotate input/output types where inference needs help.
+
+> **Passing data between nodes?** Read the **`execution-context`** skill FIRST — it is the one rule for
+> reading `item.json`, carrying data through an OCR step, and the canonical-JSON pattern. Every workflow
+> that passes structured data between nodes depends on it.
+
+> **This is the general spine.** For a specific integration (`gmail`, `odoo`, …) or topic (reaching an
+> external system → `connect-external-systems`; OCR → `document-ai`), that skill is **authoritative** —
+> follow it over any general pattern here. This skill teaches triggers, flow control, and the builder API.
+
+**Discipline:** author straight from the one-liners in this file, then run `verify_workflow` and fix only what it flags. Don't open other skills speculatively — open one only when `verify_workflow` names a node or concept this file doesn't cover.
+
+## Name and describe every node for a non-technical reader
+
+Two authoring rules make the workflow legible to the business user who reviews it on the canvas:
+
+- **Title = the business action, not the node type.** The first constructor argument is a human label.
+  Write what the step _does_ — `"Receive new order"`, `"Tag order as received"`, `"Route high-value orders"` —
+  never `"Callback"`, `"If"`, `"MapData"`. The reader should understand the flow from titles alone.
+- **`description` = one plain-language sentence** in the node's **options** (alongside `id`). It is a
+  first-class option on EVERY node — `{ id: "...", description: "..." }` — and renders in the node
+  sidebar. One non-technical sentence: what this step does and why. **Every node gets one — no
+  exceptions.** That means the **trigger** AND every step, including the ones easy to forget under a
+  big build: the trigger / test-trigger, document / PDF-scan (OCR) nodes, AI-agent nodes, and any
+  send / notify / test-notification step. A node with no `description` is a build defect — **before
+  you call the build done, re-read the whole workflow and confirm every single node has both a
+  friendly title and a `description`.**
+
+```ts no-check
+new Callback("Tag order as received", markReceived, {
+  id: "mark-received",
+  description: "Flags the order as received so the warehouse can pick it.",
+});
+```
+
+`description` lives in the same options object as `id`. For bare-id nodes (`If`, `Filter`, `Split`,
+`Switch`, `Merge`, `Wait`, …) the last argument accepts either a bare `"id"` string OR
+an options object `{ id, description }` — pass the object so you can describe the node.
+
+## A minimal complete workflow
+
+```typescript
+import { createWorkflowBuilder, WebhookTrigger, Callback } from "@codemation/core-nodes";
+import type { Items } from "@codemation/core";
+
+type Order = { orderId: string; total: number };
+
+export default createWorkflowBuilder({ id: "wf.order-webhook", name: "Order webhook" })
+  .trigger(
+    new WebhookTrigger(
+      "Receive new order",
+      { endpointKey: "orders", methods: ["POST"] },
+      undefined, // default handler
+      { id: "receive-order", description: "Accepts an order POSTed by the shop and starts the flow." },
+    ),
+  )
+  .then(
+    new Callback(
+      "Tag order as received",
+      (items: Items<Order>) => {
+        return items.map((item) => ({ json: { ...item.json, received: true } }));
+      },
+      { id: "log-order", description: "Marks each incoming order as received." },
+    ),
+  )
+  .build();
+```
+
+## The fluent DSL
+
+- `createWorkflowBuilder({ id, name })` → builder. `id` must be a `WorkflowId` (a `wf.*` string is fine).
+- `.trigger(triggerConfig)` → cursor. Exactly one trigger, first.
+- `.then(nodeConfig)` → appends a step; the chain's item type becomes that node's output.
+- `.when(true, [steps]).when(false, [steps])` (boolean form) → branches off an `If` node's `"true"`/`"false"` ports. **It auto-merges when you continue the chain:** a following `.then(...)` / `.humanApproval(...)` connects EVERY branch tail into the next node (the same fan-in the object form produces). End it with `.build()` when the branch is the end of the flow.
+- The **object form** `.when({ true: [...], false: [...] })` is the inline alternative → it merges both branch tails into one chainable cursor with a precise merged item type. Reach for it when you want the continuation typed exactly. (Keep only one branch alive — e.g. "drop non-orders, continue orders"? Use `.route({ true: (b) => b.then(...), false: () => undefined })`: a factory returning `undefined` is excluded from the merge.)
+- `.build()` → validates node ids (throws `WorkflowDefinitionError` on empty/duplicate) and returns the definition.
+
+**Filter vs If — choose this up front, it's the most common rewrite.** To KEEP ONLY the items that match a condition and DROP the rest — "only real orders", "only paid invoices", "skip anything that isn't an order" — use a **`Filter`** node (see "Per-item transforms" below): it's ONE node, the dropped items simply don't continue, and the chain stays a single trunk. Use **`If` + `.when`** ONLY when the true and false items do DIFFERENT downstream work (true → send for approval, false → auto-approve). Rule of thumb: if one side would just drop the item and the other continues, that is a `Filter`, not an `If`. Reaching for `If` to express "keep only X" and then continuing the trunk is the classic shape that has to be rewritten into a `Filter` — start with the `Filter`.
+
+Branch off an `If` with `.when`. Each step's predicate/mapper sees an `Item<T>` — read `item.json`:
+
+```typescript
+import { createWorkflowBuilder, ManualTrigger, If, Callback } from "@codemation/core-nodes";
+import type { Item, Items } from "@codemation/core";
+
+type Invoice = { amount: number };
+
+export default createWorkflowBuilder({ id: "wf.approve", name: "Approve invoice" })
+  .trigger(new ManualTrigger<Invoice>("Start", undefined, { id: "start", description: "Run this by hand." }))
+  .then(
+    new If<Invoice>("Does it need approval?", (item: Item<Invoice>) => item.json.amount > 1000, {
+      id: "needs-approval",
+      description: "Sends invoices over €1,000 for manual sign-off.",
+    }),
+  )
+  .when(true, [
+    new Callback("Send for approval", (items: Items<Invoice>) => items, {
+      id: "send-approval",
+      description: "Routes the invoice to a reviewer.",
+    }),
+  ])
+  .when(false, [
+    new Callback("Auto-approve", (items: Items<Invoice>) => items, {
+      id: "auto-approve",
+      description: "Approves small invoices automatically.",
+    }),
+  ])
+  .build();
+```
+
+**Continuing AFTER a branch** (the common shape: branch, then a shared tail like a human-approval gate).
+Just keep chaining — a `.then(...)` / `.humanApproval(...)` after the boolean `.when(...).when(...)` chain
+auto-merges EVERY branch tail into the next node. Here a HITL gate sits on the merged trunk:
+
+```typescript
+import { createWorkflowBuilder, ManualTrigger, If, MapData, inboxApproval } from "@codemation/core-nodes";
+import type { Item } from "@codemation/core";
+
+type Order = { amount: number; priority: string };
+
+export default createWorkflowBuilder({ id: "wf.order", name: "Order intake" })
+  .trigger(new ManualTrigger<Order>("Start", undefined, { id: "start", description: "Run this by hand." }))
+  .then(
+    new If<Order>("Is this a big order?", (item: Item<Order>) => item.json.amount > 1000, {
+      id: "big-gate",
+      description: "Splits large orders from normal ones.",
+    }),
+  )
+  // Boolean branches auto-merge on continuation: BOTH tails feed the next step.
+  // (The object form `.when({ true: [...], false: [...] })` is the inline alternative
+  //  when you want the merged item type typed exactly.)
+  .when(true, [
+    new MapData<Order, Order>("Mark as high priority", (item: Item<Order>) => ({ ...item.json, priority: "high" }), {
+      id: "mark-high",
+      description: "Tags big orders as high priority.",
+    }),
+  ])
+  .when(false, [
+    new MapData<Order, Order>(
+      "Mark as normal priority",
+      (item: Item<Order>) => ({ ...item.json, priority: "normal" }),
+      {
+        id: "mark-normal",
+        description: "Tags regular orders as normal priority.",
+      },
+    ),
+  ])
+  // The merged trunk keeps flowing — `.humanApproval(node, config)` is sugar for `.then(node.create(config))`.
+  // Pass every config field (Zod defaults don't relax the TS type).
+  .humanApproval(inboxApproval, {
+    title: "Approve order?",
+    body: "Review before processing.",
+    priority: "normal",
+    timeout: "48h",
+    onTimeout: "halt",
+  })
+  .build();
+```
+
+## Core nodes — one-liners
+
+Import every node from `@codemation/core-nodes`. Each construction below is complete.
+
+### Triggers (exactly one, first)
+
+```typescript
+import { ManualTrigger, WebhookTrigger, TestTrigger, schedulePollingTrigger } from "@codemation/core-nodes";
+
+// Schedule — fires one tick item on every poll cycle (defaults to every 60s). Emits { firedAt, tick }.
+// It's a defined node: build it with `.create(config, label, { id, description })`, not `new`.
+const onSchedule = schedulePollingTrigger.create({}, "Run on schedule", {
+  id: "daily",
+  description: "Runs the flow on a recurring schedule.",
+});
+
+// Manual — for runs you start by hand; optionally seed default items (object/array in arg 2),
+// then put { id, description } in the trailing slot (use arg 2 = undefined when seeding nothing).
+const manual = new ManualTrigger<{ q: string }>("Start by hand", [{ q: "hello" }], {
+  id: "manual",
+  description: "Lets an operator run the flow on demand.",
+});
+
+// Webhook — endpointKey is the URL path segment; methods are the accepted verbs.
+// Pass a Zod schema as `inputSchema` to validate the request body. The default handler is fine,
+// so options go in the trailing (4th) argument.
+const hook = new WebhookTrigger("Receive inbound request", { endpointKey: "inbound", methods: ["POST"] }, undefined, {
+  id: "hook",
+  description: "Starts the flow when an external system calls in.",
+});
+
+// Test — one item per test case; drives the Tests tab. generateItems is an async generator.
+// TestTrigger takes `description` directly in its single options object.
+const testCases = new TestTrigger<{ subject: string }>({
+  name: "Replay sample mails",
+  description: "Feeds saved sample emails through the flow for the Tests tab.",
+  async *generateItems() {
+    yield { json: { subject: "RFQ #14" } };
+  },
+});
+```
+
+### Flow control
+
+`If` and `IsTestRun` route on ports `"true"` / `"false"` — branch them with `.when`, never a bare `.then`. `Switch` exposes one port per case key.
+
+```typescript
+import { If, Switch, Merge, NoOp, IsTestRun } from "@codemation/core-nodes";
+import type { Item } from "@codemation/core";
+
+type Doc = { kind: string; urgent: boolean };
+
+const gate = new If<Doc>("Is it urgent?", (item: Item<Doc>) => item.json.urgent, {
+  id: "urgent-gate",
+  description: "Sends urgent documents down the fast path.",
+});
+
+const route = new Switch<Doc>(
+  "Route by document type",
+  {
+    cases: ["invoice", "order"],
+    defaultCase: "other",
+    resolveCaseKey: (item: Item<Doc>) => item.json.kind,
+  },
+  { id: "route-by-kind", description: "Sends each document to the handler for its type." },
+);
+
+const merge = new Merge<Doc>(
+  "Rejoin branches",
+  { mode: "passThrough" },
+  {
+    id: "rejoin",
+    description: "Brings the branches back into one stream.",
+  },
+);
+const passthrough = new NoOp<Doc>("Checkpoint", { id: "marker", description: "A no-op marker step." });
+const testGate = new IsTestRun<Doc>("Is this a test run?", {
+  id: "test-gate",
+  description: "Routes test runs away from live side effects.",
+}); // ports "true"/"false"
+```
+
+### Data shaping
+
+Predicates and mappers receive an `Item<T>` (read `item.json`); `Aggregate`/`Split` see the whole batch / one item.
+
+```typescript
+import { MapData, Filter, Aggregate, Split } from "@codemation/core-nodes";
+import type { Item, Items } from "@codemation/core";
+
+type Row = { price: number; tags: string[] };
+
+const enrich = new MapData<Row, Row & { withVat: number }>(
+  "Add VAT to each line",
+  (item: Item<Row>) => ({ ...item.json, withVat: item.json.price * 1.21 }),
+  { id: "add-vat", description: "Adds 21% VAT to every row." },
+);
+
+const keep = new Filter<Row>("Keep cheap items only", (item: Item<Row>) => item.json.price < 100, {
+  id: "cheap-only",
+  description: "Drops anything priced €100 or more.",
+});
+
+const total = new Aggregate<Row, { sum: number }>(
+  "Total the prices",
+  (items: Items<Row>) => ({ sum: items.reduce((acc, i) => acc + i.json.price, 0) }),
+  { id: "sum-prices", description: "Adds all the row prices into one total." },
+);
+
+const fanout = new Split<Row, string>("Split into one item per tag", (item: Item<Row>) => item.json.tags, {
+  id: "per-tag",
+  description: "Emits one item for each tag on a row.",
+});
+```
+
+### Fan-out & fan-in (important)
+
+A node **fans out** when its `execute` returns an **array**: the engine emits **one item per element**, so
+every later `.then(...)` runs **once per element** (zero elements ⇒ zero items downstream — branch on
+"no matches"). This is automatic — you do **not** wrap the result yourself. Because of this, a fan-out
+node must declare its output type as the **per-item element**, not the array: `.then()` carries that
+element type forward as the next node's `item.json`. (e.g. a search node that returns rows declares its
+output as one row; downstream `item.json` is one row.)
+
+- **`Split<T, E>`** — fan out an array that lives **inside** one item's json (`item.json.tags`), vs a node
+  whose `execute` returns the array directly. Both produce one item per element.
+- **`Aggregate<T, O>`** — **fan in**: collapse the whole batch back to a single item (sum, group, build one payload).
+- `.then(...)` does **not** auto-unwrap — the chain's item type is whatever the upstream node _declares_ as its
+  per-item output. If a node declares the array as its output, nothing chains cleanly after it; declare the element.
+
+### HTTP
+
+`HttpRequest` calls an external API. For `body.kind: "json"` pass the object in `body.data` (encoded once). Declare auth with `credentialSlot` — that auto-wires the bindable slot (see Credentials). Output is response metadata (`{ status, ok, json, text, ... }`), not the input item.
+
+```typescript
+import { HttpRequest } from "@codemation/core-nodes";
+
+const post = new HttpRequest("Create the record in the API", {
+  method: "POST",
+  url: "https://api.example.com/v1/records",
+  headers: { "content-type": "application/json" },
+  body: { kind: "json", data: { name: "ACME" } },
+  credentialSlot: "example",
+  responseFormat: "json",
+  id: "create-record",
+  description: "Sends the record to the external API.",
+});
+```
+
+### Glue
+
+`Callback` is the escape hatch for arbitrary per-batch logic; `Wait` pauses.
+
+```typescript
+import { Callback, Wait } from "@codemation/core-nodes";
+import type { Items } from "@codemation/core";
+
+type Msg = { text: string };
+
+const shout = new Callback<Msg>(
+  "Shout the message",
+  (items: Items<Msg>) => items.map((i) => ({ json: { text: i.json.text.toUpperCase() } })),
+  { id: "uppercase", description: "Uppercases every message." },
+);
+
+const pause = new Wait("Cool down before retry", 5000, {
+  id: "cooldown",
+  description: "Waits 5 seconds before continuing.",
+});
+```
+
+## Binary payloads in a Callback
+
+`item.binary[key]` is **metadata** (`BinaryAttachment`: id, storageKey, mimeType, size — there is **no `.data`**). Bytes come only from `ctx.binary` methods. **Never return `{ binary: { key: { data: Buffer } } }` — use `ctx.binary.attach` + `ctx.binary.withAttachment`.**
+
+Reading bytes: `ctx.binary.getJson<T>(attachment)` (also `getBytes` / `getText`) — pass the `BinaryAttachment` object, not the key string.
+
+Attaching bytes to an item: `ctx.binary.attach({ name, body, mimeType })` returns a `BinaryAttachment`; then `ctx.binary.withAttachment(item, name, att)` slots it onto the item.
+
+```typescript
+import { Callback } from "@codemation/core-nodes";
+import type { Items, NodeExecutionContext, BinaryAttachment } from "@codemation/core";
+
+type Doc = { filename: string };
+type Parsed = { wordCount: number };
+
+// Reading bytes from an upstream binary slot:
+const readBinary = new Callback<Doc, Parsed>(
+  "Count words in the file",
+  async (items: Items<Doc>, ctx: NodeExecutionContext) => {
+    return await Promise.all(
+      items.map(async (item) => {
+        const att: BinaryAttachment | undefined = item.binary?.["data"];
+        if (!att) return { json: { wordCount: 0 } };
+        const text = await ctx.binary.getText(att); // also: getBytes / getJson<T>
+        return { json: { wordCount: text.split(/\s+/).length } };
+      }),
+    );
+  },
+  { id: "count-words", description: "Counts the words in each attached file." },
+);
+
+// Attaching bytes to an item:
+const attachBinary = new Callback<Doc, Doc>(
+  "Attach the processed file",
+  async (items: Items<Doc>, ctx: NodeExecutionContext) => {
+    return await Promise.all(
+      items.map(async (item) => {
+        const body = Buffer.from("processed content");
+        const att = await ctx.binary.attach({ name: "result", body, mimeType: "text/plain" });
+        return ctx.binary.withAttachment(item, "result", att);
+        // item.binary["result"] is now a BinaryAttachment — pass it to downstream nodes
+      }),
+    );
+  },
+  { id: "attach-binary", description: "Saves the processed file onto the item." },
+);
+```
+
+## Credentials in a workflow (builder)
+
+A binding is keyed by `(workflowId, nodeId, slotKey)`, so the node that **declares** the slot is the node the credential binds to. You declare + use; the concierge binds (separate skill).
+
+**Default — `HttpRequest.credentialSlot`.** The string declares a bindable slot and authenticates the request; nothing else needed:
+
+```typescript
+import { HttpRequest } from "@codemation/core-nodes";
+
+const fetch = new HttpRequest("List contacts from the ERP", {
+  url: "https://erp.example.com/api/contacts",
+  credentialSlot: "erp", // bindable slot "erp" on this node
+  responseFormat: "json",
+  id: "list-contacts",
+  description: "Reads the contact list from the ERP.",
+});
+```
+
+**Never `fetch` in a `Callback`.** Raw `fetch`/HTTP/JSON-RPC inside a `Callback` is FORBIDDEN and the
+build gate rejects it. To call an external system, in priority order: a specialized integration node
+(e.g. the `odoo` / `gmail` nodes — search for it), an MCP-backed agent, `defineRestNode` (`rest-node`
+skill), or the `HttpRequest` node above. These declare the credential slot for you. `ctx.getCredential`
+in a `Callback` is only for non-HTTP uses (handing a token to a node SDK, signing a payload) — not for
+making the request yourself. See `connect-external-systems`.
+
+A plain `Callback` slot is **not** discoverable for binding. For credentialed custom logic that must surface a bindable slot, author a `defineNode` node with a `credentials` map (see the custom-node-development skill) — that node both declares and resolves the slot.
+
+## Error handling
+
+Pass a `retryPolicy` to retry transient failures; an unhandled `throw` fails the run. `verify_workflow` is the build gate — write, verify, fix what it names, repeat.
+
+```typescript
+import { HttpRequest } from "@codemation/core-nodes";
+import type { RetryPolicySpec } from "@codemation/core";
+
+const retry: RetryPolicySpec = { kind: "fixed", maxAttempts: 3, delayMs: 1000 };
+
+const flaky = new HttpRequest(
+  "Ping the flaky API",
+  { url: "https://api.example.com/ping", id: "flaky-api", description: "Pings an API that sometimes fails." },
+  retry,
+);
+```
+
+## One realistic complete example
+
+Schedule → authenticated GET via the **HttpRequest node** (never `fetch`) → fan the response array into one
+item per lead → shape each. To WRITE each lead back, add another `HttpRequest` (or a `defineRestNode`) —
+not a hand-rolled `fetch`. Each node has a stable explicit `id`, a business-action title, and a
+plain-language `description`.
+
+```typescript
+import { createWorkflowBuilder, schedulePollingTrigger, HttpRequest, Split, MapData } from "@codemation/core-nodes";
+import type { HttpRequestOutputJson } from "@codemation/core-nodes";
+import type { Item } from "@codemation/core";
+
+type Lead = { id: string; email: string };
+
+export default createWorkflowBuilder({ id: "wf.sync-leads", name: "Sync leads on a schedule" })
+  .trigger(
+    schedulePollingTrigger.create({}, "Run on schedule", {
+      id: "nightly",
+      description: "Runs the lead sync on a recurring schedule.",
+    }),
+  )
+  // Authenticated request via the HttpRequest node — it declares the "crm" credential slot for you.
+  .then(
+    new HttpRequest("Fetch new leads from the CRM", {
+      url: "https://crm.example.com/api/leads",
+      credentialSlot: "crm",
+      responseFormat: "json",
+      id: "fetch-leads",
+      description: "Pulls the latest leads from the CRM.",
+    }),
+  )
+  // Fan out the response array into one item per lead.
+  .then(
+    new Split<HttpRequestOutputJson, Lead>(
+      "Split into one item per lead",
+      (item: Item<HttpRequestOutputJson>) => (item.json.json as Lead[]) ?? [],
+      { id: "per-lead", description: "Processes each lead on its own." },
+    ),
+  )
+  // Shape each lead. To upsert it, add another HttpRequest (method: "PUT") or a defineRestNode — not fetch.
+  .then(
+    new MapData<Lead, Lead & { queued: boolean }>(
+      "Mark each lead as queued",
+      (item: Item<Lead>) => ({ ...item.json, queued: true }),
+      { id: "queue-upsert", description: "Flags each lead ready to write back." },
+    ),
+  )
+  .build();
+```
+
+## Gotchas
+
+- **Unique, stable node ids.** Set an explicit `id` on every node. Without one, the engine slugifies the `name`, so two same-named nodes collide and `.build()` throws — and renaming a label re-keys credential bindings.
+- **Friendly title + `description` on every node.** Title = the business action a non-tech reader recognizes; `description` = one plain sentence in the node's options (`{ id, description }`). Both render in the canvas / sidebar.
+- **Per-item execution.** Nodes run once per item in the batch. `MapData`/`Filter`/`If`/`Split` see one `Item<T>`; `Callback`/`Aggregate` see the whole `Items<T>`.
+- **`If` / `IsTestRun` route on ports.** Branch their `"true"`/`"false"` ports with `.when`, not a bare `.then`. `Switch` ports are its case keys.
+- **Binary payloads never go on `item.json`.** Store bytes via `ctx.binary.attach(...)` and pass the `Item.binary` slot through — never base64 on JSON.
+- **Boolean `.when` branches auto-merge on continuation.** `.when(true,[...]).when(false,[...]).then(next)` connects EVERY branch tail into `next` (the same fan-in the object form gives). End on `.build()` when the branch is the flow's end. The continued cursor is typed as the pre-branch item type — use the **object form** `.when({ true:[...], false:[...] })` when you need the merged type typed exactly, or `.route({...})` to drop a branch (return `undefined`).
+- **Branches must rejoin compatible types.** When two `.when` branches feed a shared downstream node, both must emit the type that node accepts.
+- **Talking to an external system?** Follow the routing hierarchy in `connect-external-systems` — prefer a specialized node, MCP server, or `defineRestNode` over a hand-rolled `fetch` in a `Callback`.