@codemation/agent-skills 0.4.0 → 0.5.1

package/skills/builder/execution-context/SKILL.md

@@ -0,0 +1,436 @@
+---
+name: execution-context
+description: Teaches how data moves between nodes in Codemation workflows — read the immediate prior node's output from item.json (the engine put it there), read ANY earlier node's output directly from its source via ctx.data.getOutputItem<T>(NODE_ID, idx) (you never carry it forward), and reserve ctx.binary for actual file bytes. Essential for any multi-step workflow (order intake, document processing, ETL) where a later step needs a value that an intermediate node replaced. Read this before writing any node that reads data from a prior step.
+compatibility: Applies to all Codemation workflow nodes and the Callback, MapData, defineNode, and itemExpr APIs.
+tags: execution, context, item, json, binary, mapdata, data-passing, anti-pattern, ocr, workflow, authoring, nodes, between-steps, getoutputitem, cross-step-read, order-intake, document-processing
+uses: "@codemation/core-nodes, @codemation/core"
+---
+
+# Execution context — how data moves between nodes
+
+The engine passes every node's output to the **immediate next** node as `item.json`. That covers the
+common case. But many workflows need a value that lives **further back** — the trigger's `messageId`,
+an early extraction field — after an intermediate node (a structured-output `AIAgent`, an OCR scanner)
+**replaced** `item.json`. For that, you do not carry the value forward through every step. You read it
+**directly from its source node** with `ctx.data.getOutputItem<T>(NODE_ID, idx)`.
+
+**Core rules:**
+
+1. **Immediate prior node's output → `item.json`.** The previous node already put its result there.
+   Read it directly.
+2. **Any EARLIER node's output → `ctx.data.getOutputItem<T>(NODE_ID, idx)`.** Fetch it from the SOURCE
+   node by id. It does not matter what an intermediate agent did to `item.json` in between — the source
+   node's output is still recorded in the run and you read it from there. No carry-forward, no re-merge.
+3. **`ctx.binary` is for file bytes only.** Binary attachments (PDFs, images, uploaded files) live in
+   `item.binary` and are read via `ctx.binary`. Never stash structured JSON as a fabricated binary to
+   "carry it along" — structured data belongs on `item.json`.
+
+## Reading an earlier node's output — `ctx.data.getOutputItem`
+
+`RunDataSnapshot` (available as `ctx.data` inside any node) exposes the recorded output of every node
+that has already run this turn:
+
+```ts no-check
+getOutputItem<TJson = unknown>(nodeId: NodeId, itemIndex: number, output?: OutputPortKey): Item<TJson> | undefined
+```
+
+- `nodeId` — the **id you gave the source node** (the trigger, an extraction step, …). `NodeId` is a
+  plain `string`, so a stable `const` id (e.g. `const TRIGGER_ID = "gmail-trigger"`) passes directly.
+- `itemIndex` — the index into that node's output, aligned to the **current item's lineage**. For 1:1
+  flows pass the current index — often the loop index `idx`, or `0` at a single-item point such as the
+  trunk start. Under fan-out (one upstream item produced many, or vice versa) the indices no longer line
+  up 1:1, so align carefully.
+- Returns `Item<TJson> | undefined` — always guard with `?.json` and a fallback.
+- `output` defaults to `"main"`; omit it for the common case.
+
+### Recover a clobbered field from its source node
+
+A structured-output `AIAgent` (e.g. an Odoo sync) **replaces** `item.json` with its schema-typed object
+— the trigger's `messageId` / email metadata is GONE from `item.json` after it. You do **not** thread
+that metadata through every node. You read it back from the **trigger** directly:
+
+```typescript
+import { z } from "zod";
+import { createWorkflowBuilder, AIAgent, CodemationChatModelConfig } from "@codemation/core-nodes";
+import { OnNewGmailTrigger, ReplyToGmailMessage } from "@codemation/core-nodes-gmail";
+import { itemExpr } from "@codemation/core";
+import type { OnNewGmailTriggerItemJson } from "@codemation/core-nodes-gmail";
+
+// Stable ids so getOutputItem can back-reference the source node by id.
+const TRIGGER_ID = "gmail-trigger";
+const SYNC_ID = "odoo-sync";
+
+const model = new CodemationChatModelConfig("Managed AI", "medium");
+
+// The Odoo-sync agent's output schema — this REPLACES item.json. No messageId here.
+const SyncOutput = z.object({ saleOrderId: z.string(), unmatched: z.array(z.string()) });
+type SyncOutputT = z.infer<typeof SyncOutput>;
+
+export default createWorkflowBuilder({ id: "wf.order-sync-reply", name: "Order sync + reply" })
+  .trigger(
+    new OnNewGmailTrigger("New order email", {
+      mailbox: "me",
+      labelIds: ["orders"],
+      downloadAttachments: true,
+    }),
+  )
+  // Step 1: structured-output agent. Its { saleOrderId, unmatched } REPLACES item.json —
+  // the trigger's messageId is no longer on item.json after this point.
+  .then(
+    new AIAgent<OnNewGmailTriggerItemJson, SyncOutputT>({
+      name: "Odoo sync agent",
+      id: SYNC_ID,
+      messages: [{ role: "system", content: "Sync the order to Odoo. Respond with strict JSON." }],
+      chatModel: model,
+      outputSchema: SyncOutput,
+    }),
+  )
+  // Step 2: reply to the sender. messageId is NOT on item.json anymore — read it straight
+  // from the TRIGGER's recorded output via getOutputItem. The current item is the sync output.
+  .then(
+    new ReplyToGmailMessage(
+      "Reply to sender",
+      {
+        // ✅ Read messageId from the SOURCE node (the trigger), not from item.json.
+        messageId: itemExpr(({ ctx }) => {
+          const triggerItem = ctx.data.getOutputItem<OnNewGmailTriggerItemJson>(TRIGGER_ID, 0);
+          return triggerItem?.json.messageId ?? "";
+        }),
+        text: itemExpr(({ item }) => {
+          const sync = item.json as SyncOutputT; // current item.json IS the sync output
+          return `Your order has been processed (Odoo SO: ${sync.saleOrderId}).`;
+        }),
+      },
+      "reply-to-sender",
+    ),
+  )
+  .build();
+```
+
+### The same read inside a `Callback` (item-index aligned)
+
+Inside a `Callback` you iterate items, so the `idx` from the loop is the lineage index to pass — it
+aligns the current item with the matching trigger item in a 1:1 flow:
+
+```typescript
+import { Callback } from "@codemation/core-nodes";
+import type { Items } from "@codemation/core";
+import type { OnNewGmailTriggerItemJson } from "@codemation/core-nodes-gmail";
+import type { DocScannerOutput } from "@codemation/core-nodes";
+
+const TRIGGER_ID = "gmail-trigger";
+
+// The OCR scanner already replaced item.json with { markdown, fields }. We need the email
+// subject + body from the trigger to feed the next agent — read them from the trigger by id.
+type RouterInput = DocScannerOutput & {
+  mailSubject?: string;
+  mailBody?: string;
+  messageId: string;
+};
+
+// ✅ Read the trigger item by id, using the loop idx as the lineage index (1:1 flow).
+const mergeMailBody = new Callback<DocScannerOutput, RouterInput>(
+  "Merge mail body into item",
+  (items: Items<DocScannerOutput>, ctx) =>
+    items.map((item, idx) => {
+      const triggerItem = ctx.data.getOutputItem<OnNewGmailTriggerItemJson>(TRIGGER_ID, idx);
+      return {
+        ...item,
+        json: {
+          ...item.json,
+          mailSubject: triggerItem?.json.subject,
+          mailBody: triggerItem?.json.textPlain ?? triggerItem?.json.snippet,
+          messageId: triggerItem?.json.messageId ?? "",
+        } satisfies RouterInput,
+      };
+    }),
+  { id: "merge-mail-body" },
+);
+export {};
+```
+
+The key point: the **graph between the trigger and this node is irrelevant**. An OCR step, an agent, a
+`Switch` — whatever ran in between and however it reshaped `item.json` — the trigger's output is still
+recorded under `TRIGGER_ID`, and `getOutputItem` reads it from there.
+
+## Anti-patterns vs. good patterns
+
+### (a) A recovery cast for a field that was clobbered — read it from the source instead
+
+When a structured-output agent or OCR node replaces `item.json`, a field you still need (e.g.
+`messageId`) is no longer there. The **broken** fix is a phantom cast that pretends the field survived —
+it compiles but is `undefined` at runtime because the field was clobbered and never re-fetched. The
+clean fix is to read it from its source node with `getOutputItem`.
+
+```typescript
+import { z } from "zod";
+import { Callback } from "@codemation/core-nodes";
+import type { Items } from "@codemation/core";
+import type { OnNewGmailTriggerItemJson } from "@codemation/core-nodes-gmail";
+
+const TRIGGER_ID = "gmail-trigger";
+
+const SyncOutput = z.object({ saleOrderId: z.string() });
+type SyncOutputT = z.infer<typeof SyncOutput>;
+// The agent's output replaced item.json — messageId is NOT on it. The phantom cast below
+// pretends otherwise: it type-checks but `messageId` is `undefined` at runtime.
+type SyncOutputWithMessageId = SyncOutputT & { messageId: string };
+
+// ❌ WRONG — phantom recovery cast. messageId was clobbered by the agent and never re-fetched,
+// so item.json.messageId is undefined at runtime even though TypeScript is satisfied.
+const badRecover = new Callback<SyncOutputT, { ack: string }>(
+  "Acknowledge",
+  (items: Items<SyncOutputT>) =>
+    items.map((item) => {
+      const clobbered = item.json as SyncOutputWithMessageId; // ← lie: messageId isn't there
+      return { json: { ack: `ack for ${clobbered.messageId}` } }; // undefined at runtime
+    }),
+  { id: "bad-recover" },
+);
+
+// ✅ CORRECT — read messageId from its SOURCE node (the trigger) via getOutputItem.
+const goodRecover = new Callback<SyncOutputT, { ack: string }>(
+  "Acknowledge",
+  (items: Items<SyncOutputT>, ctx) =>
+    items.map((item, idx) => {
+      const triggerItem = ctx.data.getOutputItem<OnNewGmailTriggerItemJson>(TRIGGER_ID, idx);
+      return { json: { ack: `ack for ${triggerItem?.json.messageId ?? "unknown"}` } };
+    }),
+  { id: "good-recover" },
+);
+export {};
+```
+
+### (b) Re-deriving the same shape repeatedly instead of reading the source
+
+Don't recompute a value from raw inputs in every node. Compute it once and read it where you need it —
+either from the immediate prior node's `item.json`, or, for an earlier node, from its source via
+`getOutputItem`.
+
+```typescript
+import { createWorkflowBuilder, ManualTrigger, MapData, Callback } from "@codemation/core-nodes";
+import type { Items, Item } from "@codemation/core";
+
+type Raw = { companyName: string; vatId: string; lines: Array<{ sku: string; qty: number }> };
+type Canonical = { company: string; vat: string; lineCount: number };
+
+// ❌ WRONG — deriving the same shape twice, in two separate nodes.
+const badDerive1 = new Callback<Raw, { tag: string }>(
+  "Tag order",
+  (items: Items<Raw>) =>
+    items.map((item) => ({
+      json: { tag: `${item.json.companyName}-${item.json.lines.length}` }, // ← derives from raw again
+    })),
+  { id: "bad-derive-1" },
+);
+
+// ✅ GOOD — one MapData shapes the value once; the next node reads THAT shape from item.json.
+const canonical = new MapData<Raw, Canonical>(
+  "Canonicalize",
+  (item: Item<Raw>) => ({
+    company: item.json.companyName,
+    vat: item.json.vatId,
+    lineCount: item.json.lines.length,
+  }),
+  { id: "canonical" },
+);
+
+// Reads item.json.company / item.json.lineCount — never re-derives from raw.
+const tagOrder = new Callback<Canonical, { tag: string }>(
+  "Tag order",
+  (items: Items<Canonical>) => items.map((item) => ({ json: { tag: `${item.json.company}-${item.json.lineCount}` } })),
+  { id: "tag-order" },
+);
+export {};
+```
+
+### (c) JSON stashed inside a fabricated binary instead of on item.json
+
+```typescript
+import { createWorkflowBuilder, ManualTrigger, Callback } from "@codemation/core-nodes";
+import type { Items, NodeExecutionContext } from "@codemation/core";
+
+type Order = { orderId: string; total: number };
+
+// ❌ WRONG — encoding structured JSON as bytes and attaching it as a binary is wasted ceremony.
+// item.json already carries structured data between nodes at zero cost.
+const badBinaryStash = new Callback<Order, Order>(
+  "Stash to binary",
+  async (items: Items<Order>, ctx: NodeExecutionContext) => {
+    const results = [];
+    for (const item of items) {
+      // Attaching structured JSON as a binary → retrieving it in the next node via ctx.binary.getJson
+      // is needless overhead when the engine just passes item.json for free.
+      const att = await ctx.binary.attach({
+        name: "order",
+        body: Buffer.from(JSON.stringify(item.json)),
+        mimeType: "application/json",
+      });
+      results.push(ctx.binary.withAttachment(item, "order", att));
+    }
+    return results;
+  },
+  { id: "bad-binary-stash" },
+);
+
+// ✅ GOOD — structured data belongs on item.json. The engine carries it for you.
+// Use ctx.binary ONLY for real file bytes (PDFs, images, CSVs from readWorkspaceFileNode, etc.).
+const passThrough = new Callback<Order, Order>(
+  "Pass order",
+  (items: Items<Order>) => items.map((item) => ({ json: item.json })),
+  { id: "pass-order" },
+);
+export {};
+```
+
+## MapData — narrow/shape an extraction output (not a cross-step carry)
+
+A `MapData` is still a good tool to **narrow or shape** a noisy extraction output (`{ markdown, fields }`
+from OCR, or a raw agent object) into the tight shape the **immediate next** step wants. That is its job:
+a clean local transform. It is **not** required for cross-step survival — a downstream node that needs an
+earlier node's data should `getOutputItem` it from the source, not depend on it having been threaded
+through a canonical shape.
+
+```typescript
+import {
+  createWorkflowBuilder,
+  WebhookTrigger,
+  MapData,
+  Callback,
+  codemationDocumentScannerNode,
+} from "@codemation/core-nodes";
+import type { Item, Items } from "@codemation/core";
+import type { DocScannerOutput } from "@codemation/core-nodes";
+
+// A tight shape for the immediate next step — narrowed from the noisy OCR output.
+type OrderFields = { vendor: string; totalAmount: number };
+
+export default createWorkflowBuilder({ id: "wf.invoice-intake", name: "Invoice intake" })
+  .trigger(new WebhookTrigger("Receive invoice", { endpointKey: "invoice-upload", methods: ["POST"] }))
+  // Step 1: OCR — replaces item.json with { markdown, fields }.
+  .then(codemationDocumentScannerNode.create({ analyzerType: "invoice" }, "Scan invoice", "scan-invoice"))
+  // Step 2: MapData NARROWS the noisy OCR output into the shape the next step needs. This is a
+  // local convenience for the immediate next node — NOT a carry-forward of earlier-node data.
+  .then(
+    new MapData<DocScannerOutput, OrderFields>(
+      "Narrow OCR fields",
+      (item: Item<DocScannerOutput>) => ({
+        vendor: (item.json.fields["VendorName"]?.value as string | undefined) ?? "Unknown",
+        totalAmount: (item.json.fields["InvoiceTotal"]?.value as number | undefined) ?? 0,
+      }),
+      { id: "narrow-fields" },
+    ),
+  )
+  // Step 3: the immediate next node reads the narrowed shape from item.json.
+  .then(
+    new Callback<OrderFields, { summary: string }>(
+      "Summarize",
+      (items: Items<OrderFields>) =>
+        items.map((item) => ({ json: { summary: `${item.json.vendor}: €${item.json.totalAmount}` } })),
+      { id: "summarize" },
+    ),
+  )
+  .build();
+```
+
+## ctx.binary — when to use it
+
+`ctx.binary` is the API for **file bytes**. Use it when:
+
+- A `readWorkspaceFileNode`, Gmail attachment, or webhook upload put bytes on `item.binary`.
+- You need to read a PDF, image, CSV, or JSON file that arrived as a binary attachment.
+
+Never use it as a workaround for passing structured data between nodes. If it's JSON, it belongs on
+`item.json` (immediate prior) or is read via `getOutputItem` (earlier node).
+
+```typescript
+import { Callback } from "@codemation/core-nodes";
+import type { NodeExecutionContext } from "@codemation/core";
+
+type FileMeta = { fileId: string; binarySlot: string };
+type Product = { sku: string; price: number };
+
+// ✅ Correct: ctx.binary reads actual file bytes from a slot an upstream node attached.
+const parseFile = new Callback<FileMeta, { count: number }>(
+  "Parse product file",
+  async (items, ctx: NodeExecutionContext) => {
+    const results: Array<{ json: { count: number } }> = [];
+    for (const item of items) {
+      const attachment = item.binary?.["data"]; // bytes put there by readWorkspaceFileNode
+      if (!attachment) throw new Error('No binary at slot "data"');
+      const products = await ctx.binary.getJson<Product[]>(attachment); // bounded read + parse
+      results.push({ json: { count: products.length } });
+    }
+    return results;
+  },
+  { id: "parse-file" },
+);
+export {};
+```
+
+### (d) Enrichment Callback REPLACES item.json — always spread the existing payload
+
+A `Callback`'s return value **replaces `item.json` entirely** — exactly like any other node.
+An enrichment step (one that looks up a derived field, e.g. a matched ERP partner id) that
+returns ONLY the new field DISCARDS the entire prior payload (companyName, lineItems,
+contactEmail, …) that the **immediate next** node needs.
+
+**The rule:** an enrichment `Callback` MUST return `{ ...item.json, newField }` — not just
+`{ newField }`. (If an even earlier node's value is needed downstream, that's `getOutputItem`'s job,
+not this spread's.)
+
+```typescript
+import { Callback } from "@codemation/core-nodes";
+import type { Items } from "@codemation/core";
+
+type OrderCanonical = {
+  companyName: string;
+  contactEmail: string;
+  lineItems: Array<{ sku: string; qty: number }>;
+};
+type WithPartner = OrderCanonical & { partnerId: number; partnerName: string };
+
+// ❌ WRONG — returns only the looked-up fields, replacing item.json and discarding
+// companyName, contactEmail, lineItems that the next node needs.
+const badEnrich = new Callback<OrderCanonical, { partnerId: number; partnerName: string }>(
+  "Enrich partner",
+  (items: Items<OrderCanonical>) =>
+    items.map((i) => ({
+      json: { partnerId: 42, partnerName: i.json.companyName }, // ← DISCARDS the rest
+    })),
+  { id: "bad-enrich" },
+);
+
+// ✅ CORRECT — spreads the entire existing payload, THEN adds the new fields.
+// The next node still sees companyName, contactEmail, lineItems, AND the new ids.
+const goodEnrich = new Callback<OrderCanonical, WithPartner>(
+  "Enrich partner",
+  (items: Items<OrderCanonical>) =>
+    items.map((i) => ({
+      json: { ...i.json, partnerId: 42, partnerName: i.json.companyName },
+    })),
+  { id: "good-enrich" },
+);
+export {};
+```
+
+## Summary
+
+| Question                                                              | Answer                                                                                                           |
+| --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| Where is the IMMEDIATE prior node's output?                           | `item.json` — read it directly.                                                                                  |
+| How do I read an EARLIER node's output (trigger, an extraction step)? | `ctx.data.getOutputItem<T>(NODE_ID, idx)` — from the source node by id. You never carry it forward.              |
+| A field I need was clobbered by an intermediate agent/OCR — now what? | Read it from its source node via `getOutputItem`. Never a phantom recovery cast — that's `undefined` at runtime. |
+| When do I need a MapData?                                             | To NARROW/shape an extraction output for the IMMEDIATE next step. Not required for cross-step survival.          |
+| When do I use `ctx.binary`?                                           | Only for file bytes (PDF, image, CSV, JSON file from a read node).                                               |
+| Can I JSON-encode structured data into a binary to pass it along?     | No — that's waste. `item.json` carries it for free; earlier nodes via `getOutputItem`.                           |
+| My enrichment Callback looks up a field — what do I return?           | `{ ...item.json, newField }` — spread first, add second. Never only `{ newField }`.                              |
+
+## Read next when needed
+
+- `document-ai` — the OCR node that produces `{ markdown, fields }` (one source you may later `getOutputItem`).
+- `ai-agent` — extraction via LLM; `outputSchema` produces a typed object that REPLACES `item.json`.
+- `workflow-dsl` — `Callback`, `MapData`, `itemExpr`, and the full builder API.
+- `connect-external-systems` → "Build what works; report what's missing" — if a required value (label
+  name, folder ID, record ID) is unknown at build time, use a named TODO placeholder and
+  `record_decision`; never fabricate an identifier.

package/skills/{codemation-framework-concepts → builder/framework-concepts}/SKILL.md

@@ -1,21 +1,13 @@
 ---
-name: codemation-framework-concepts
-description: Explains Codemation package boundaries, runtime concepts, observability shape, and the normal consumer mental model. Use when the user asks where code belongs across `@codemation/core`, `@codemation/host`, `@codemation/next-host`, `@codemation/cli`, workflows, plugins, credentials, activation, telemetry, or runtime modes. Read this first when starting any Codemation task — it points at the right skill for the work.
-compatibility: Designed for Codemation apps, plugins, and framework contributors.
+name: framework-concepts
+description: Explains Codemation package boundaries, runtime concepts, and the consumer-versus-framework mental model across @codemation/core, @codemation/host, @codemation/next-host, and @codemation/cli. Use to orient on where code belongs before picking a more specific skill.
 tags: concepts, architecture
 ---
 
 # Codemation Framework Concepts
 
-## Mental model
-
 Codemation is a workflow engine with a layered package structure. `@codemation/core` owns the engine and runtime contracts (must stay pure — no HTTP, UI, or vendor SDKs). `@codemation/host` adds persistence, credentials, APIs, and scheduler wiring. `@codemation/next-host` is the framework UI shell. `@codemation/cli` runs local development, build, and serve. Consumer apps define behavior in `codemation.config.ts` and `src/workflows/` — they never touch core internals.
 
-## When to use / when NOT
-
-Use this skill to orient on package ownership, runtime shape, observability boundaries, and the consumer/framework divide.
-Do not use as a substitute for detailed CLI, workflow DSL, or plugin implementation guidance when you already know which skill you need.
-
 ## Core concepts
 
 - **workflows** define behavior; **triggers** start runs; **nodes** process items; **items** carry `item.json` data.
@@ -27,14 +19,14 @@ Do not use as a substitute for detailed CLI, workflow DSL, or plugin implementat
 
 ## Where to go next
 
-| Task                                  | Skill                                |
-| ------------------------------------- | ------------------------------------ |
-| Authoring workflows                   | `codemation-workflow-dsl`            |
-| Building a reusable node              | `codemation-custom-node-development` |
-| Building a credential type            | `codemation-credential-development`  |
-| Packaging as a plugin                 | `codemation-plugin-development`      |
-| Calling an MCP server from a workflow | `codemation-mcp-capabilities`        |
-| CLI commands / dev loop               | `codemation-cli`                     |
+| Task                                  | Skill                     |
+| ------------------------------------- | ------------------------- |
+| Authoring workflows                   | `workflow-dsl`            |
+| Building a reusable node              | `custom-node-development` |
+| Building a credential type            | `credential-development`  |
+| Packaging as a plugin                 | `plugin-development`      |
+| Calling an MCP server from a workflow | `mcp-capabilities`        |
+| CLI commands / dev loop               | `cli`                     |
 
 ## Read next when needed