@codemation/agent-skills 0.1.9 → 0.2.0

package/CHANGELOG.md

@@ -1,5 +1,47 @@
 # @codemation/agent-skills
 
+## 0.2.0
+
+### Minor Changes
+
+- 8285ec0: Add `codemation-ai-agent-node` skill teaching the coding agent the `AIAgent` constructor signature, message shape, output contract (`{ output: string }`), and the four managed model IDs currently in the allowlist. Also covers BYOK (`OpenAIChatModelConfig`) and MCP tool attachment.
+- 8285ec0: Add `codemation-mcp-capabilities` skill: documents the control-plane `GET /api/registry/capabilities` endpoint, MCP server credential kinds, and the workflow-author flow for discovering and binding MCP servers. Originally produced under Story 14 in a workspace-local install; relocated here so `pnpm codemation skills sync` picks it up.
+
+### Patch Changes
+
+- 8285ec0: docs(codemation-workflow-dsl): add "Verifying your workflow" section pointing agents to use `verify_workflow` MCP tool instead of `pnpm typecheck`.
+- 8285ec0: Reorganize agent skills for faster coding-agent start-up (Story C).
+  - `codemation-framework-concepts` promoted to router skill: adds "Where to go next" section mapping tasks to skills, and updated frontmatter describing it as the index to read first.
+  - `codemation-custom-node-development` split into a slim TOC (53 lines) plus three new focused refs: `define-node-per-item.md`, `define-batch-node.md`, `credential-aware-nodes.md`. Binary/HTTP/MS-Graph patterns consolidated into the existing `node-patterns.md`.
+
+- 8285ec0: Remove dead references to `describe_node` and `list_nodes` from workflow-dsl and ai-agent-node skills; replace with `find_examples` as the discovery surface.
+- 8285ec0: Sprint 11 Story D: deepen `find_examples` as the canonical discovery surface in skills.
+  - Expand `codemation-workflow-dsl` "Discovering nodes and patterns" section with why examples are canonical, when to call first, two query patterns, zero-hit response (ask user or adapt with confirmation), and anti-pattern (don't grep node_modules).
+  - Strengthen `codemation-ai-agent-node` call-to-action with concrete `find_examples` query suggestions for AIAgent patterns.
+
+- 8285ec0: docs(codemation-workflow-dsl): note that search results include install-state fields (`installed`, `requiresInstall`) and how agents should use them to plan `install_package` calls.
+- 8285ec0: Sprint 12 Story C: add self-solving fallback chain to `codemation-workflow-dsl` skill.
+  - Add "When no example matches — the self-solving fallback chain" section after "Discovering nodes and patterns".
+  - Four-tier chain: retry with intent variations → defineRestNode (HTTP APIs) → HttpRequest (inline one-off) → defineNode (non-HTTP custom logic).
+  - Explicit "What NOT to do" and "Surfacing what you did" sub-sections.
+  - Agent never asks the non-technical user for fallback choices; picks per the chain and reports what it used.
+
+- 8285ec0: Sprint 14 coverage: tests for WhenBuilder DSL helper, InMemoryWorkflowExecutionRepository retention paths, DevTrackedProcessTreeKiller edge cases, ConsumerCliTsconfigPreparation resolution, ListenPortConflictDescriber ss fallback, RedisRunEventBus publish/subscribe/teardown, CodemationChatModelFactory HMAC signing, registerCoreNodes smoke, single-react-component-per-file rule branches, and CodemationAgentSkillsCli error/help paths. No production code changes.
+- 8285ec0: docs(skills): add complete workflow example to workflow-dsl skill
+- 8285ec0: Fix workflow-dsl skill docs: `workflow("id")` is manual-trigger only. Cron, webhook, and other non-manual triggers must use `createWorkflowBuilder({id, name}).trigger(new XxxTrigger(...))` and chain with `.then(new SomeNodeConfig(...))` — the fluent `.map`/`.if`/`.agent`/`.node` sugar is only available via `manualTrigger(...)`.
+
+  Affected: `codemation-workflow-dsl/SKILL.md`, `codemation-workflow-dsl/references/builder-patterns.md`, `codemation-workflow-dsl/references/complete-example.md`, `codemation-mcp-capabilities/references/agent-with-mcp.ts`.
+
+## 0.1.10
+
+### Patch Changes
+
+- [#114](https://github.com/MadeRelevant/codemation/pull/114) [`ec985a3`](https://github.com/MadeRelevant/codemation/commit/ec985a3264696b421e8be7c84c7cead6a85cbe6c) Thanks [@cblokland90](https://github.com/cblokland90)! - Fix `pnpm create codemation <name>` failing with `ENOENT … node_modules/agent-skills/skills` when dlx'd from npm.
+
+  `@codemation/agent-skills`'s `exports` field only declared `.`, so `require.resolve("@codemation/agent-skills/package.json")` was blocked by Node's exports gate. `create-codemation`'s resolver fell back to a workspace-only relative path that doesn't exist outside the monorepo. Adds `./package.json` and `./skills/*` to the exports map so subpath access works for consumers — and bumps `create-codemation` patch so the next release pins the fixed agent-skills version.
+
+- [#110](https://github.com/MadeRelevant/codemation/pull/110) [`4902978`](https://github.com/MadeRelevant/codemation/commit/49029782243ece59ab6aa5bb46396db445cad47c) Thanks [@cblokland90](https://github.com/cblokland90)! - Add per-package `test:unit` scripts so Turbo can address each package individually for affected-only filtering. No runtime changes — dev-tooling only.
+
 ## 0.1.9
 
 ### Patch Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@codemation/agent-skills",
-  "version": "0.1.9",
+  "version": "0.2.0",
   "description": "Reusable agent skills for Codemation projects and plugin development.",
   "publishConfig": {
     "access": "public"
@@ -19,7 +19,9 @@
       "types": "./lib/agent-skills-extractor.d.ts",
       "import": "./lib/agent-skills-extractor.mjs",
       "default": "./lib/agent-skills-extractor.mjs"
-    }
+    },
+    "./package.json": "./package.json",
+    "./skills/*": "./skills/*"
   },
   "bin": {
     "codemation-agent-skills": "./bin/codemation-agent-skills.mjs"
@@ -46,6 +48,7 @@
   ],
   "scripts": {
     "changeset:verify": "pnpm --workspace-root run changeset:verify",
-    "test": "vitest run"
+    "test": "vitest run",
+    "test:unit": "vitest run"
   }
 }

package/skills/codemation-ai-agent-node/SKILL.md

@@ -0,0 +1,128 @@
+---
+name: codemation-ai-agent-node
+description: AIAgent constructor, message shape, output contract, and chat-model configs (managed and BYOK). Read before writing any workflow that uses AIAgent.
+compatibility: Codemation core-nodes. Requires @codemation/core-nodes import.
+---
+
+# Codemation AI Agent Node
+
+> **Start here: call `find_examples` before reading further.**
+>
+> - `find_examples({ query: "AIAgent" })` — basic usage and constructor patterns
+> - `find_examples({ query: "AIAgent multi-step" })` — chained pipeline patterns
+> - `find_examples({ query: "AIAgent tools" })` — agent with callable tools
+> - `find_examples({ query: "AIAgent gmail classify" })` — domain-specific examples
+>
+> The sections below are a quick orientation for when you need the exact constructor or output shape.
+
+## Use this skill when
+
+Writing a workflow that uses `AIAgent` — classification, extraction, summarisation, drafting, decision, or any step that calls an LLM.
+Use `codemation-workflow-dsl` for the surrounding workflow structure.
+Use `codemation-mcp-capabilities` when the agent needs MCP servers.
+
+## When to use `AIAgent` vs other approaches
+
+Use `AIAgent` when an item needs an LLM call with a fixed or per-item prompt and optional tool use.
+Use a plain `Callback` instead when the logic is deterministic code (no LLM needed).
+Use the `.agent(...)` fluent helper on a manual-trigger workflow only if you need the full fluent chain sugar — under the hood it also produces an `AIAgent`.
+
+## Constructor
+
+```ts
+import { AIAgent } from "@codemation/core-nodes";
+
+new AIAgent({
+  name: string,                          // display name and default node id slug
+  messages: AgentMessageConfig,          // see below
+  chatModel: ChatModelConfig,            // see Managed and BYOK sections below
+  tools?: ReadonlyArray<ToolConfig>,     // optional callable tools
+  id?: string,                           // stable node id (set explicitly if node has credential bindings)
+})
+```
+
+## `messages` shape
+
+`messages` is an ordered array of `{ role, content }` objects.
+
+```ts
+messages: [
+  { role: "system", content: "You are a helpful assistant that classifies emails." },
+  { role: "user", content: (args) => `Classify this email:\n\n${args.item.json.body}` },
+];
+```
+
+- `role` is `"system"` | `"user"` (use `"assistant"` only for few-shot examples — rare).
+- `content` can be a plain string or a function `(args: { item, itemIndex, items, ctx }) => string`.
+- Put the detailed instructions in the `system` message and the per-item data in the `user` message.
+
+## Output shape
+
+`AIAgent` emits `{ output: string }` on its single port `main`.
+
+The next node sees `item.json.output` as the agent's text response.
+Type your downstream `Callback` accordingly:
+
+```ts
+.then(new Callback<{ output: string }>("Handle result", (item) => {
+  const reply = item.json.output;
+  // ...
+}))
+```
+
+If you set `outputSchema` (a Zod schema), the agent validates and parses the output into a structured object. Without `outputSchema`, `item.json.output` is always a plain string.
+
+## Managed model (no credentials needed)
+
+```ts
+import { AIAgent, CodemationChatModelConfig } from "@codemation/core-nodes";
+
+new AIAgent({
+  name: "Classify email",
+  messages: [
+    { role: "system", content: "Classify the email as spam or not-spam." },
+    { role: "user", content: (args) => args.item.json.body as string },
+  ],
+  chatModel: new CodemationChatModelConfig(
+    "Claude Haiku", // display label
+    "anthropic/claude-haiku-4-5-20251001", // managed model id
+  ),
+});
+```
+
+### Currently allowlisted managed models
+
+| Model id                              | Notes                |
+| ------------------------------------- | -------------------- |
+| `anthropic/claude-haiku-4-5-20251001` | Fastest and cheapest |
+| `anthropic/claude-sonnet-4-6`         | Balanced             |
+| `anthropic/claude-opus-4-5-20251101`  | High capability      |
+| `anthropic/claude-opus-4-6`           | Latest flagship      |
+
+Discover live: `GET <CONTROL_PLANE_URL>/api/llm/managed-models`
+
+## BYOK model (user supplies their own key)
+
+```ts
+import { AIAgent, OpenAIChatModelConfig } from "@codemation/core-nodes";
+
+new AIAgent({
+  name: "Summarise",
+  id: "summarise-agent", // stable id — required when node has a credential binding
+  messages: [
+    { role: "system", content: "Summarise the following text in one paragraph." },
+    { role: "user", content: (args) => args.item.json.text as string },
+  ],
+  chatModel: new OpenAIChatModelConfig(
+    "OpenAI GPT-4o", // display label
+    "gpt-4o", // OpenAI model id
+    "openai", // credential slot key — matches the slot used in getCredentialRequirements
+  ),
+});
+```
+
+`OpenAIChatModelConfig` requires the user to connect an `openai.apiKey` credential. The concierge handles credential acquisition — the coding agent must not invent credentials.
+
+## MCP servers
+
+If you need tools / MCP servers on the agent, see the `codemation-mcp-capabilities` skill.

package/skills/codemation-credential-development/SKILL.md

@@ -12,6 +12,12 @@ Use this skill for defining new credential types, wiring them into apps or plugi
 
 Do not use this skill for general workflow authoring unless credential slots or runtime sessions are the core problem.
 
+## Credential binding stability
+
+Credentials bind to a node via `(workflowId, nodeId, slotKey)`. The `nodeId` defaults to a slug of the node's `name` label (lowercase, non-alphanumeric runs replaced with `-`). Renaming a credential-using node's label silently changes its id and the binding appears unbound in the UI — the operator must re-attach manually.
+
+To prevent this: either keep the node's label stable across edits, or set an explicit `id:` on the node config so the id is decoupled from the label.
+
 ## Core mental model
 
 1. A credential type defines public config, secret material, session creation, and health testing.

package/skills/codemation-credential-development/references/credential-patterns.md

@@ -1,5 +1,20 @@
 # Credential Patterns
 
+## Node id and binding stability
+
+A credential binding is stored as `(workflowId, nodeId, slotKey)`. The `nodeId` for each workflow node defaults to a slug of its `name` label. Changing the label changes the id, and the previously configured binding appears unbound.
+
+For production workflows with credential-using nodes, prefer an explicit `id:` on the node config:
+
+```ts
+.node("Fetch from API", MyApiNodeConfig, {
+  id: "fetch-from-api", // stable across label renames
+  credentials: { apiKey: myApiCredential },
+})
+```
+
+Without an explicit `id:`, keep the node's label constant or plan to re-bind after a rename.
+
 ## Standard shape
 
 Use `defineCredential(...)` to declare:
@@ -40,6 +55,34 @@ Optional or power-user fields (for example custom OAuth scopes) can be tucked be
 
 See **`packages/core/docs/credential-ui-fields.md`** in the repository root layout.
 
+## OAuth2 credentials (URL-template variant)
+
+For credentials that go through the OAuth2 redirect flow (Microsoft Graph, Slack, GitHub, Notion, etc.), declare the authorize and token URLs directly on the credential's `auth` definition. The host's `OAuth2ProviderRegistry` substitutes `{publicFieldKey}` placeholders from the credential's public config at connect time (URL-encoded).
+
+```ts
+auth: {
+  kind: "oauth2",
+  // providerId is a free-form label for telemetry / DB rows / Better Auth provider naming.
+  // It is NOT used for any registry lookup — URLs come from the fields below.
+  providerId: "microsoft",
+  authorizeUrl: "https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/authorize",
+  tokenUrl: "https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token",
+  scopes: ["openid", "offline_access", "User.Read", "Mail.Read"],
+},
+```
+
+Three `auth` variants exist:
+
+1. **URL-template (preferred for new plugins).** Carries `authorizeUrl` / `tokenUrl` / optional `userInfoUrl` directly with `{fieldKey}` substitution. Self-contained — adding a new provider needs no core or host edits.
+2. **Built-in `providerId` shortcut.** Only `google` is recognized; kept for backwards compatibility. Do not add new providers here.
+3. **`providerFromPublicConfig`.** URLs read verbatim from public field values at runtime. Rare; the template variant covers almost every real case more ergonomically.
+
+Notes for plugin authors:
+
+- Host stores post-callback OAuth material with snake_case keys (`access_token`, `refresh_token`, `expiry`, `scope`, `token_type`). Read those keys inside `createSession` / `test`, NOT camelCase.
+- The redirect URI returned to providers rewrites loopback IPs (`127.0.0.1`, `[::1]`) to `localhost` so Azure AD (AADSTS50011) and other providers with the same restriction accept it.
+- The default `Mail.Read` (and similar single-mailbox) Microsoft scopes only cover the credential owner. To monitor a shared mailbox via `/users/{upn}/...`, request `Mail.Read.Shared` (delegated) or admin-consented application permissions.
+
 ## Health and activation
 
 - deploy the workflow and credential type

package/skills/codemation-custom-node-development/SKILL.md

@@ -12,27 +12,42 @@ Use this skill for reusable custom node work, whether the node lives inside an a
 
 Do not use this skill for pure workflow chaining questions unless the node implementation itself is changing.
 
-## Default approach
+## Per-item vs batch
 
-1. Start with `defineNode(...)`.
-2. Implement **`execute(args, context)`** — one mapped **input** in, one output payload per item (activations are still batch-shaped; the engine iterates items for you).
-3. Give the node a stable key and a clear title.
-4. Optionally set **`icon`** on the `defineNode` definition so the workflow canvas shows a proper glyph (same string contract as `NodeConfigBase.icon`).
-5. Use **`defineBatchNode(...)`** with **`run(items, context)`** only when the node must process the **entire batch** at once (legacy batch semantics).
-6. Promote callback-heavy logic into a node when the graph or tests need a stronger boundary.
+**`defineNode(...)` (per-item)** — the engine calls `execute(args, context)` once per item. This is the right default for the vast majority of nodes: straightforward logic, credential slots, input schema, optional fan-out.
 
-## Node rules
-
-1. Prefer helper-based nodes first.
-2. Keep nodes deterministic and focused.
-3. Request credentials through named slots instead of hard-coded secrets.
-4. Put **static** options (credentials, retry policy, labels) on **config**; put **per-item** behavior in **inputs** / wire JSON and optional **`itemExpr`** on config fields (consistent with built-in nodes).
-5. Drop to class-based node APIs only when you need constructor-injected collaborators, decorators, or deeper runtime metadata.
+**`defineBatchNode(...)` (batch)** — the engine calls `run(items, context)` with the full activation batch. Use only when the node genuinely needs to see all items at once (aggregation, bulk API calls, cross-item correlation).
 
-## Testing with `WorkflowTestKit`
+When in doubt, start with `defineNode`.
 
-For engine-backed tests without the host, use **`WorkflowTestKit`** from **`@codemation/core/testing`**: **`registerDefinedNodes([...])`**, then **`runNode`** or **`run`**. See the plugin development doc and `@codemation/core` tests for examples.
+## Node rules
 
-## Read next when needed
+1. Keep nodes deterministic and focused.
+2. Request credentials through named slots — never hard-code secrets.
+3. Put **static** options (credentials, retry policy, labels) on **config**; put **per-item** behavior in **inputs** / wire JSON and optional `itemExpr` on config fields.
+4. **Emit files with `ctx.binary`, not base64 in `json`** — base64 in `item.json` bloats persisted run data. See `references/node-patterns.md`.
+5. Drop to class-based node APIs only when you need constructor-injected collaborators, decorators, or deeper runtime metadata.
 
-- Read `references/node-patterns.md` for `defineNode(...)` patterns and packaging guidance.
+## Minimal `defineNode` example
+
+```ts
+import { defineNode } from "@codemation/core";
+import { z } from "zod";
+
+export const uppercaseNode = defineNode({
+  key: "example.uppercase",
+  title: "Uppercase field",
+  icon: "lucide:languages",
+  inputSchema: z.object({ field: z.string() }),
+  async execute({ input }) {
+    return { ...input, field: input.field.toUpperCase() };
+  },
+});
+```
+
+## Read next
+
+- `references/define-node-per-item.md` — full `defineNode(...)` contract, `inputSchema`, `itemExpr`, fan-out, assertion nodes, and `WorkflowTestKit` usage. Load this when writing or debugging a per-item node.
+- `references/define-batch-node.md` — `defineBatchNode(...)` contract and when to choose batch over per-item. Load this when the node must see the entire batch at once.
+- `references/credential-aware-nodes.md` — credential slots, typed sessions, and how to test credential-aware nodes. Load this when your node needs a credential.
+- `references/node-patterns.md` — binary payloads (`ctx.binary`, `attach`, `withAttachment`), fan-out return shapes, polling-trigger binary patterns, MS Graph attachment download, and HTTP binary round-trips. Load this when working with file data or HTTP binaries.

package/skills/codemation-custom-node-development/references/credential-aware-nodes.md

@@ -0,0 +1,38 @@
+# Credential-Aware Nodes
+
+Load this when your node needs a typed credential (OAuth token, API key, or any `defineCredential(...)` type) injected at runtime.
+
+## Core rule
+
+Request credentials through **named slots** on the node config instead of hard-coding secrets. The framework resolves the slot to a live typed session at execution time.
+
+## Adding a credential slot to `defineNode`
+
+```ts
+import { defineNode } from "@codemation/core";
+import { myApiCredentialType } from "./myApiCredential.js";
+
+export const callApiNode = defineNode({
+  key: "example.call-api",
+  title: "Call My API",
+  credentials: {
+    api: myApiCredentialType, // slot name → credential type
+  },
+  async execute({ input }, { credentials }) {
+    const session = await credentials.api.getSession();
+    // session is typed by myApiCredentialType.sessionSchema
+    const response = await fetch("https://api.example.com/data", {
+      headers: { Authorization: `Bearer ${session.accessToken}` },
+    });
+    return response.json();
+  },
+});
+```
+
+## Typed sessions
+
+`credentials.<slot>.getSession()` returns the shape declared in the credential type's `sessionSchema`. The framework handles refresh, storage, and error propagation — your node only consumes the session.
+
+## Testing credential-aware nodes
+
+Supply a mock credential in `WorkflowTestKit` rather than live credentials. See `codemation-credential-development` for the full `defineCredential(...)` story, typed sessions, and credential testing patterns.

package/skills/codemation-custom-node-development/references/define-batch-node.md

@@ -0,0 +1,38 @@
+# Define Batch Node
+
+Load this when you need to author a `defineBatchNode(...)` node that processes all items in one call.
+
+## When to use `defineBatchNode` instead of `defineNode`
+
+- The node must see the **entire activation batch** at once (e.g. an aggregation, a bulk API call, or a node that correlates items against each other).
+- Legacy batch semantics are required by the calling workflow.
+- You need the same contract as built-in batch-shaped nodes such as `Aggregate`.
+
+For the common case (one-item-at-a-time logic), prefer `defineNode` — the engine handles iteration for you.
+
+## Minimal skeleton
+
+```ts
+import { defineBatchNode } from "@codemation/core";
+import { z } from "zod";
+
+export const sumNode = defineBatchNode({
+  key: "example.sum",
+  title: "Sum numeric field",
+  inputSchema: z.object({ value: z.number() }),
+  async run(items, { config }) {
+    const total = items.reduce((acc, item) => acc + (item.json as { value: number }).value, 0);
+    return [{ json: { total } }];
+  },
+});
+```
+
+## Contract
+
+- `run(items, context)` receives the **full array** of activation items.
+- Return an array of output items (same length as input is not required — you can fan-in to one, or fan-out to many).
+- The context object exposes `config`, `credentials`, and `execution` (same as `defineNode`).
+
+## Advanced fallback
+
+Reach for class-based node APIs when constructor-injected collaborators are required, plugin packaging needs the lower-level runtime contract, or decorators/persisted metadata need tighter control.

package/skills/codemation-custom-node-development/references/define-node-per-item.md

@@ -0,0 +1,61 @@
+# Define Node (Per-Item)
+
+Load this when you need to author a `defineNode(...)` node that processes one item at a time.
+
+## When to use `defineNode`
+
+- Node logic is straightforward.
+- The node belongs to one app or plugin package.
+- Helper-based credential slots are sufficient.
+- You do not need to inspect the entire batch in one call.
+
+## Minimal skeleton
+
+```ts
+import { defineNode } from "@codemation/core";
+import { z } from "zod";
+
+export const uppercaseNode = defineNode({
+  key: "example.uppercase",
+  title: "Uppercase field",
+  icon: "lucide:languages", // optional — Lucide, builtin:, si:, or image URL
+  inputSchema: z.object({ field: z.string() }),
+  async execute({ input, item }, { config }) {
+    return { ...input, [config.field]: String(input.field).toUpperCase() };
+  },
+});
+```
+
+## Contract
+
+- `execute(args, context)` is called **once per item** by the engine.
+- Return a plain JSON object → one output item on `main`.
+- Return a top-level array → **fan-out** (one item per element).
+- Return `emitPorts({ portName: [...] })` for multi-port routing.
+- Return an item-shaped `{ json, binary?, meta?, paired? }` when you need explicit binary/meta control.
+
+## Config fields with `itemExpr`
+
+Place **static** options (credentials, retry policy, labels) on `config`; place **per-item** values in `inputs` using `itemExpr` on config fields — consistent with built-in nodes.
+
+## Input schema and `inputSchema`
+
+Supply `inputSchema` (Zod) to get typed `input` in `execute` and to drive the canvas form. The engine validates items against it before calling `execute`.
+
+## Testing with `WorkflowTestKit`
+
+```ts
+import { createEngineTestKit, registerDefinedNodes } from "@codemation/core/testing";
+
+const kit = createEngineTestKit();
+registerDefinedNodes([uppercaseNode]);
+const result = await kit.runNode(uppercaseNode, { json: { field: "hello" } });
+```
+
+Use `WorkflowTestKit` from `@codemation/core/testing` for engine-backed tests without the host.
+
+## Custom assertion nodes
+
+Set `emitsAssertions: true` on the node config to record results into `TestSuiteRun` infrastructure. The host's `TestSuiteRunTracker` listens for `nodeCompleted` events on runs with `ctx.testContext` set and persists each emitted item (matching `AssertionResult`) as a `TestAssertion` row.
+
+Per-item nodes can also read `ctx.testContext?.{testSuiteRunId, testCaseIndex}` to branch on test mode — useful for synthetic outputs or skipping irreversible side effects.

package/skills/codemation-custom-node-development/references/node-patterns.md

@@ -1,5 +1,7 @@
 # Node Patterns
 
+Load this when working with file data, binary payloads, HTTP binaries, MS Graph attachments, or when you need reference on fan-out return shapes and polling-trigger binary patterns.
+
 ## Start here
 
 Use `defineNode(...)` when:
@@ -55,3 +57,165 @@ Reach for class-based node APIs when:
 - **`defineNode`** runs **`execute` once per item** (with optional **`inputSchema`** and **`itemExpr`** on config fields before **`execute`**)
 - **`defineBatchNode`** runs **`run`** once per activation batch
 - keep nodes deterministic and testable; prefer real code paths or in-memory collaborators over heavy mocking
+
+## Emitting items, fan-out, and binaries (for AI codegen)
+
+**Return shapes**
+
+- Return **plain JSON** → one output item with that **`json`** (unless the value is a **top-level array**, which **fans out** to one item per element).
+- Return **`emitPorts({ portName: [...] })`** for multi-port routing.
+- Return an **item-shaped** `{ json, binary?, meta?, paired? }` when you need explicit **`binary`** / **`meta`** / **`paired`** control.
+
+**Never put bulk file content in `item.json`**
+
+- Fields like `contentBase64`, `data`, or multi-megabyte strings are stored **inside persisted run / step JSON** in the database. That **scales poorly** (base64 is larger than raw bytes) and hurts snapshots and tooling.
+- **Correct:** `const attachment = await args.ctx.binary.attach({ name: "file", body: bytesOrStream, mimeType, filename })` then `return args.ctx.binary.withAttachment({ json: { ok: true } }, "file", attachment)` (or build `{ json, binary }` by hand).
+- **`body`** types match **`BinaryBody`**: `Uint8Array`, `ArrayBuffer`, `ReadableStream`, or async iterable of chunks (same idea as **`HttpRequest`** downloading a body).
+- **`keepBinaries: true`** only **preserves existing** **`item.binary`** through a plain JSON return; it does **not** convert base64 strings in **`json`** into attachments.
+
+**Triggers**
+
+- Emit **one `Item` per external record**; use **`item.binary`** per record for files—not one item whose **`json`** contains an array of embedded files.
+- For polling triggers that fetch records carrying file payloads (mail attachments, message media, etc.), do this in two phases:
+  1. In `runCycle` (the polling step), fetch only the **metadata** (id, name, contentType, size). The result is persisted into the trigger's setup state and into emitted item JSON, so it must stay small.
+  2. In `execute(items, ctx)`, when the cfg opts into downloads, fetch each blob's bytes from the source API and register them via `ctx.binary.attach(...)`. Then return items via `ctx.binary.withAttachment(item, slot, stored)`.
+- **Do not** request the full payload in the polling fetch (e.g. Microsoft Graph `$expand=attachments` returns base64 `contentBytes` inline; use `$expand=attachments($select=id,name,contentType,size)` to keep the response light). Large polling responses bloat the run state on every cycle, even when no item is emitted.
+
+## Binary payloads in sub-workflow chains
+
+Binary slots attached inside a node survive SubWorkflow boundaries with no extra work. The shared `BinaryStorage` DI singleton means `ctx.binary.openReadStream` works regardless of which run originally stored the bytes.
+
+### Pattern: attach in a node, read in the parent after SubWorkflow
+
+```ts
+// Child node — attaches a slot and returns the modified item.
+export const parseAndStoreNode = defineNode({
+  key: "example.parse-store",
+  title: "Parse and Store",
+  inputSchema: z.object({ filename: z.string() }),
+  async execute({ input, item }, { binary }) {
+    const bytes = Buffer.from("...parsed content...");
+    const att = await binary.attach({
+      name: "parsed",
+      body: bytes,
+      mimeType: "text/plain",
+      filename: `${input.filename}.txt`,
+    });
+    return binary.withAttachment(item, "parsed", att);
+  },
+});
+```
+
+After `SubWorkflowNode` returns, the parent's continuation nodes see `item.binary["parsed"]` and can call `ctx.binary.openReadStream(item.binary["parsed"])` to read the bytes.
+
+### Testing binary across SubWorkflow with `WorkflowTestKit`
+
+```ts
+import { DefaultExecutionContextFactory, InMemoryBinaryStorage } from "@codemation/core";
+import { createEngineTestKit } from "@codemation/core/testing";
+import { ItemHarnessNodeConfig } from "@codemation/core/testing";
+
+const storage = new InMemoryBinaryStorage();
+const kit = createEngineTestKit({
+  executionContextFactory: new DefaultExecutionContextFactory(storage),
+});
+
+// Use ItemHarnessNodeConfig (NOT CallbackNodeConfig) for nodes that must modify items:
+const attachNode = new ItemHarnessNodeConfig(
+  "Attach",
+  z.unknown(),
+  async ({ item, ctx }) => {
+    const att = await ctx.binary.attach({
+      name: "doc",
+      body: Buffer.from("content"),
+      mimeType: "application/pdf",
+      filename: "doc.pdf",
+    });
+    return ctx.binary.withAttachment(item as Item, "doc", att);
+  },
+  { id: "attach" },
+);
+// CallbackNodeConfig is fine for assertion-only (observe) nodes — it echoes input unchanged.
+```
+
+Important: `CallbackNodeConfig` discards its callback return value and always echoes input items. Never use it for nodes that must attach binary or transform items.
+
+## MS Graph: selective attachment download
+
+Use `OutlookAttachmentDownload` from `@codemation/core-nodes-msgraph` when you have already obtained attachment metadata (filename, contentType, id) and want to download only specific attachments.
+
+```ts
+import { onNewMsGraphMailTrigger, outlookAttachmentDownloadNode } from "@codemation/core-nodes-msgraph";
+
+workflow("wf.download-resumes")
+  .trigger(onNewMsGraphMailTrigger, { mailbox: "me", folderId: "inbox" })
+  .then(
+    outlookAttachmentDownloadNode.create(
+      {
+        messageId: "", // falls back to item.json when empty
+        attachmentId: "", // falls back to item.json when empty
+        binarySlot: "resume",
+        sizeCapBytes: 10 * 1024 * 1024,
+      },
+      "DownloadResume",
+    ),
+  )
+  .build();
+```
+
+Key constraints:
+
+- Only `#microsoft.graph.fileAttachment` is supported — `itemAttachment` / `referenceAttachment` throw immediately.
+- Set `keepBinaries: true` on any downstream node that needs to pass the binary slot forward.
+- The credential is `msGraphMailOAuthCredentialType`; `Mail.Read` scope is sufficient.
+
+## HTTP + binary: download to a slot, then upload from a slot
+
+`HttpRequest` (from `@codemation/core-nodes`) natively handles binary response and request bodies.
+
+### Download a file to a binary slot
+
+```ts
+import { HttpRequest } from "@codemation/core-nodes";
+import { workflow } from "@codemation/host";
+
+export default workflow("wf.download-pdf")
+  .manualTrigger<{ url: string }>("Start", { url: "" })
+  .then(
+    new HttpRequest("DownloadResume", {
+      responseFormat: "binary",
+      responseBinarySlot: "resume", // default is "response"
+      responseSizeCapBytes: 10 * 1024 * 1024, // 10 MiB cap (default 100 MiB)
+    }),
+  )
+  .build();
+// item.json gets: { status, headers, binarySlot, contentType, size, filename? }
+// item.binary["resume"] holds the BinaryAttachment reference — never base64.
+```
+
+### Upload binary bytes from a slot
+
+```ts
+new HttpRequest("UploadResume", {
+  method: "POST",
+  url: "https://api.example.com/files",
+  body: { kind: "binary", slot: "resume" },
+  // Content-Type defaults to the attachment's mimeType.
+});
+```
+
+### Download then upload (full round-trip)
+
+```ts
+export default workflow("wf.mirror-pdf")
+  .manualTrigger<{ sourceUrl: string; targetUrl: string }>("Start", { sourceUrl: "", targetUrl: "" })
+  .then(new HttpRequest("Download", { urlField: "sourceUrl", responseFormat: "binary", responseBinarySlot: "file" }))
+  .then(new HttpRequest("Upload", { urlField: "targetUrl", method: "PUT", body: { kind: "binary", slot: "file" } }))
+  .build();
+```
+
+Key rules:
+
+- Never put bytes or base64 in `item.json` — always use `ctx.binary`.
+- `responseSizeCapBytes` is checked against `Content-Length` before reading the body; set it for untrusted sources.
+- Use `keepBinaries: true` on downstream nodes that must forward the slot.